On-line or Downloadable Papers
ECD_overview. Brief overview of evidence-centered assessment design, from Frase, L.T.,
Chudorow, M., Almond, R.G., Burstein, J., Kukich, K., Mislevy, R.J., Steinberg,
L.S., & Singley, K. (in press). Technology and assessment. In H.F. O'Neil
& R. Perez (Eds.), Technology applications in assessment: A learning view.
[html document]
FourProcess. Almond, R.G., Steinberg, L.S., & Mislevy, R.J. (1999). A sample assessment
using the four process framework. White paper prepared for the IMS Working
Group on Question and Test Inter-Operability. Princeton, NJ: Educational Testing
Service. [pdf format]
WhereNumbersComeFrom. Mislevy, R.J., Almond, R.G., Yan, D., & Steinberg, L.S. (1999). Bayes nets
in educational assessment: Where do the numbers come from? In K.B. Laskey
& H.Prade (Eds.), Proceedings of the Fifteenth Conference on Uncertainty
in Artificial Intelligence (437-446). San Francisco: Morgan Kaufmann.
[preprint] [pdf format]
ProbReasoning. Mislevy, R.J., & Gitomer, D.H. (1996). The role of probability-based inference
in an intelligent tutoring system. User-Modeling and User-Adapted Interaction,
5, 253-282. [preprint] [pdf format]
Leverage. Mislevy, R.J., Steinberg, L.S., Almond, R.G., Haertel, G., & Penuel, W.
(in press). Leverage points for improving educational assessment. In B. Means
& G. Haertel (Eds.), Evaluating the effects of technology in education.
MakingSense. Mislevy, R.J., Steinberg, L.S., Breyer, F.J., Almond, R.G., & Johnson, L.
(in press). Making sense of data from complex assessment. Applied Measurement
in Education. [preprint] [pdf format]
MarketBasket Paper. Prepared for NAS Conference on marketbasket reporting in Washington D.C.,
February 6-7, 2000. Updated version includes examples and figures. [pdf format
-- 94 pages]
Linking Educational Assessments: Concepts, Issues, Methods, and Prospects. This paper describes the basic concepts of linking educational assessments.
Although it discusses statistical machinery for interpreting evidence about
students' achievements, this paper's principal message is that linking tests
is not just a technical problem. Technical questions can not be asked or answered
until questions about the nature of learning and achievement and about the
purposes and consequences of assessment are addressed. Some fundamental ideas
about educational assessment and test theory (chains of inferences and roles
of judgment) are considered. The following approaches to linking assessments
are described: (1) equating in physical measurement and educational assessment;
(2) calibration in physical measurement and educational assessment; (3) projection
in physical measurement and educational assessment; and (4) statistical moderation
and social moderation. Implications for a system of monitoring progress toward
educational standards are discussed. 1992. [pdf format -- about 90 pages]
Linking Adult Literacy Tests. 1995 paper commissioned by the Department of Labor and the Department of
Education, on prospects, methods, and issues on the topic of linking results
across the several literacy assessments that are used to evaluate adult literacy
programs. [pdf format -- about 60 pages]
Conditional Probability Models. "Models for condtional probability tables in educational assessment," by
R.G. Almond, L. Dibello, F. Jenkins, R.J. Mislevy, L.S. Steinberg, and D.
Yan. Paper presented at the 2001 AI in Statistics conference. Concerns developing
Bayesian priors based on subject matter experts' knowledge about the structure
and the difficulty of complex assessment tasks. [pdf format -- 7 pages]
Psychometric Principles. “Psychometric Principles in Student Evaluation,” by R.J. Mislevy, M.R. Wilson,
K. Ercikan, & N. Chudowsky. 9/30/01 draft of a chapter to appear in D. Stufflebeam
& T. Kellaghan (Eds.), International Handbook of Educational Evaluation.
Dordrecht, the Netherlands: Kluwer Academic Press. [pdf format – 68 pages]
Modeling conditional probabilities.
“Modeling conditional probabilities in complex educational assessments,”
by R.J. Mislevy, D. Senturk, R.G. Almond, L.V. Dibello, F. Jenkins, L.S. Steinberg,
& D. Yan. Paper presented at the International Meeting of the Psychometric
Society, July 15-19, 2001, Osaka, Japan. "Effective theta" method for expressing
experts' knowledge in Bayesian prior distributions, MCMC estimation to refine
parameters, and illustrations from the Biomass project. Extensions of Almond
et al. AIStat 2001 paper. [pdf format -- 56 pages]
Task-Based Language Assessment. "Design and analysis in task-based language assessment," by Robert J. Mislevy,
Linda S. Steinberg, and Russell G. Almond. To appear in a special issue of
Language Assessment, John Norris, guest editor. [pdf format – 32 pages]
Metropolis sampling. "Metropolis Sampling for a 2-Valued Random Variable." Heuristic arguments
for why Metropolis sampling works, and the rationale for "burn-in" cycles.
Revised 12/8/01. [pdf format -- 5 pages]
Structure of Assessments.
Preprint draft of "On the structure of educational assessments," by Robert
J. Mislevy, Linda S. Steinberg, and Russell G. Almond. To appear in Measurement.
[pdf format – 84 pages]
Biomass Overview.
The Biomass project at ETS produced a running prototype of a web-delivered,
standards-based assessment of science inquiry skills. It demonstrates the
use of an evidence-centered approach to assessment design, and the implementation
of a four process delivery system. [pdf format – 37 pages]
Delivery Architecture & Assessment Design. Almond, R.G., Steinberg, L.S., & Mislevy, R.J. (2002). "A Four-Process Architecture
for Assessment Delivery, with Connections to Assessment Design. " To appear
in the Journal of Learning Technology and Assessment. Lays out connections
between the models of the ECD assessment design framework and the processes
of the four-process delivery system architecture. [pdf format -- 67 pages]
Argument Substance and Argument Structure in Educational Assessment. To appear in Law, Probability, and Risk. Presented at the Conference
on Inference, Culture, and Ordinary Thinking in Dispute Resolution, Benjamin
N. Cardozo School of Law, Yeshiva University, New York, New York, April 27-29,
2003. [pdf format -- 39 pages]
Intuitive Test Theory. Draft of "Intuitive Test Theory," by Robert J. Mislevy & Henry I. Braun.
This paper is the text of an invited address by the first author at the Annual
Dinner Meeting of the Princeton Association for Computing Machinery (ACM)
and Institute of Electrical and Electronics Engineers (IEEE) Computer Society
Chapters, in Kingston, NJ, May 22, 2003.
Brief Intro to ECD.
By Robert J. Mislevy, Russell G. Almond, and Janice Lukas. A brief introduction
to "evidence centered assessment design" (ECD). Includes a glossary and links
to web material for further reading.
Correspondence concerning normalized gain scores. Two pieces of correspondence motivated by a posting by Prof. Richard Hake on the AERA-D listserv at <http://lists.asu.edu/cgi-bin/wa?A2=ind0603&L=aera-d&T=0&F=&S=&P=339> concerning the use of normalized gain scores <g> to measure learning. Implications of Evidence-Centered Design for Educational Testing.
Preprint draft of "On the structure of educational assessments," by Robert
J. Mislevy & Geneva D. Haertel. To appear in Educational M easurement: Issues and Practice.
On the Roles of External Knowledge Representations in Assessment Design.
Draft technical report by Robert J. Mislevy, John T. Behrens, Randy E. Bennett,
Sarah F. Demark, Dennis C. Frezzo, Roy Levy, Daniel H. Robinson, Daisy
Wise Rutstein, Valerie J. Shute, Ken Stanley, Fielding I. Winters.
Evidentiary
foundations of mixture item response theory models. By Robert Mislevy,
Roy Levy, Marc Kroopnick, and Daisy Rutstein.
A
Taxonomy of Adaptive Testing: Opportunities and Advantages of Online Assessment.
By Roy Levy, John Behrens, and Robert Mislevy.Structural,
Functional and Semiotic Symmetries in Simulation-based Games and Assessments.
By Robert Mislevy, John Behrens, Dennis Frezzo, Marc Kroopnick, and Daisy
Rutstein. Technology
Supports for Assessment Design. By Robert J. Mislevy, Isaac I. Bejar,
Randy E. Bennett,Some
Implications of Expertise Research for Educational Assessment. Keynote address
at the 34th International Association for Educational Assessment (IAEA) Conference,
Cambridge University, September 8, 2008.
How
Cognitive Science Challenges the Educational Measurement Tradition. Web
commentary for MEASUREMENT: