On-line or Downloadable Papers
ECD_overview. Brief overview of evidence-centered assessment design, from Frase, L.T.,
Chudorow, M., Almond, R.G., Burstein, J., Kukich, K., Mislevy, R.J., Steinberg,
L.S., & Singley, K. (in press). Technology and assessment. In H.F. O'Neil
& R. Perez (Eds.), Technology applications in assessment: A learning view.
[html document]
FourProcess. Almond, R.G., Steinberg, L.S., & Mislevy, R.J. (1999). A
sample assessment using the four process framework. White paper prepared
for the IMS Working Group on Question and Test Inter-Operability. Princeton,
NJ: Educational Testing Service. [pdf format]
WhereNumbersComeFrom. Mislevy, R.J., Almond, R.G., Yan, D., & Steinberg,
L.S. (1999). Bayes nets in educational assessment: Where do the numbers come
from? In K.B. Laskey & H.Prade (Eds.), Proceedings of the Fifteenth Conference
on Uncertainty in Artificial Intelligence (437-446). San Francisco: Morgan
Kaufmann. [preprint] [pdf format]
ProbReasoning. Mislevy, R.J., & Gitomer, D.H. (1996). The role of probability-based
inference in an intelligent tutoring system. User-Modeling and User-Adapted
Interaction, 5, 253-282. [preprint] [pdf format]
Leverage. Mislevy, R.J., Steinberg, L.S., Almond, R.G., Haertel, G., & Penuel,
W. (in press). Leverage points for improving educational assessment. In B. Means
& G. Haertel (Eds.), Evaluating the effects of technology in education.
MakingSense. Mislevy, R.J., Steinberg, L.S., Breyer, F.J., Almond, R.G.,
& Johnson, L. (in press). Making sense of data from complex assessment. Applied
Measurement in Education. [preprint] [pdf format]
MarketBasket Paper. Prepared for NAS Conference on marketbasket reporting
in Washington D.C., February 6-7, 2000. Updated version includes examples and
figures. [pdf format -- 94 pages]
Linking Educational Assessments: Concepts, Issues, Methods, and Prospects.
This paper describes the basic concepts of linking educational assessments.
Although it discusses statistical machinery for interpreting evidence about
students' achievements, this paper's principal message is that linking tests
is not just a technical problem. Technical questions can not be asked or answered
until questions about the nature of learning and achievement and about the purposes
and consequences of assessment are addressed. Some fundamental ideas about educational
assessment and test theory (chains of inferences and roles of judgment) are
considered. The following approaches to linking assessments are described: (1)
equating in physical measurement and educational assessment; (2) calibration
in physical measurement and educational assessment; (3) projection in physical
measurement and educational assessment; and (4) statistical moderation and social
moderation. Implications for a system of monitoring progress toward educational
standards are discussed. 1992. [pdf format -- about 90 pages]
Linking Adult Literacy Tests. 1995 paper commissioned by the Department
of Labor and the Department of Education, on prospects, methods, and issues
on the topic of linking results across the several literacy assessments that
are used to evaluate adult literacy programs. [pdf format -- about 60 pages]
Conditional Probability Models. "Models for condtional probability tables
in educational assessment," by R.G. Almond, L. Dibello, F. Jenkins, R.J. Mislevy,
L.S. Steinberg, and D. Yan. Paper presented at the 2001 AI in Statistics conference.
Concerns developing Bayesian priors based on subject matter experts' knowledge
about the structure and the difficulty of complex assessment tasks. [pdf format
-- 7 pages]
Psychometric Principles. “Psychometric Principles in Student Evaluation,”
by R.J. Mislevy, M.R. Wilson, K. Ercikan, & N. Chudowsky. 9/30/01 draft of a
chapter to appear in D. Stufflebeam & T. Kellaghan (Eds.), International
Handbook of Educational Evaluation. Dordrecht, the Netherlands: Kluwer Academic
Press. [pdf format – 68 pages]
Modeling conditional probabilities. “Modeling conditional probabilities
in complex educational assessments,” by R.J. Mislevy, D. Senturk, R.G. Almond,
L.V. Dibello, F. Jenkins, L.S. Steinberg, & D. Yan. Paper presented at the International
Meeting of the Psychometric Society, July 15-19, 2001, Osaka, Japan. "Effective
theta" method for expressing experts' knowledge in Bayesian prior distributions,
MCMC estimation to refine parameters, and illustrations from the Biomass project.
Extensions of Almond et al. AIStat 2001 paper. [pdf format -- 56 pages]
Task-Based Language Assessment. "Design and analysis in task-based language
assessment," by Robert J. Mislevy, Linda S. Steinberg, and Russell G. Almond.
To appear in a special issue of Language Assessment, John Norris, guest
editor. [pdf format – 32 pages]
Metropolis sampling. "Metropolis Sampling for a 2-Valued Random Variable."
Heuristic arguments for why Metropolis sampling works, and the rationale for
"burn-in" cycles. Revised 12/8/01. [pdf format -- 5 pages]
Structure of Assessments. Preprint draft of "On the structure of educational
assessments," by Robert J. Mislevy, Linda S. Steinberg, and Russell G. Almond.
To appear in Measurement. [pdf format – 84 pages]
Biomass Overview. The Biomass project at ETS produced a running prototype
of a web-delivered, standards-based assessment of science inquiry skills. It
demonstrates the use of an evidence-centered approach to assessment design,
and the implementation of a four process delivery system. [pdf format – 37 pages]
Delivery Architecture & Assessment Design. Almond, R.G., Steinberg, L.S.,
& Mislevy, R.J. (2002). "A Four-Process Architecture for Assessment Delivery,
with Connections to Assessment Design. " To appear in the Journal of Learning
Technology and Assessment. Lays out connections between the models of the ECD
assessment design framework and the processes of the four-process delivery system
architecture. [pdf format -- 67 pages]
Argument Substance and Argument Structure in Educational Assessment. To
appear in Law, Probability, and Risk. Presented at the Conference on
Inference, Culture, and Ordinary Thinking in Dispute Resolution, Benjamin N.
Cardozo School of Law, Yeshiva University, New York, New York, April 27-29,
2003. [pdf format -- 39 pages]
Intuitive Test Theory. Draft of "Intuitive Test Theory," by Robert J. Mislevy
& Henry I. Braun. This paper is the text of an invited address by the first
author at the Annual Dinner Meeting of the Princeton Association for Computing
Machinery (ACM) and Institute of Electrical and Electronics Engineers (IEEE)
Computer Society Chapters, in Kingston, NJ, May 22, 2003.
Brief Intro to ECD. By Robert J. Mislevy, Russell G. Almond, and Janice
Lukas. A brief introduction to "evidence centered assessment design" (ECD).
Includes a glossary and links to web material for further reading.
Correspondence concerning normalized gain scores. Two pieces of correspondence
motivated by a posting by Prof. Richard Hake on the AERA-D listserv at <http://lists.asu.edu/cgi-bin/wa?A2=ind0603&L=aera-d&T=0&F=&S=&P=339>
concerning the use of normalized gain scores <g> to measure learning. Implications
of Evidence-Centered Design for Educational Testing. Preprint draft of
"On the structure of educational assessments," by Robert J. Mislevy
& Geneva D. Haertel. To appear in Educational M easurement: Issues
and Practice.
On the Roles of External Knowledge Representations in Assessment Design.
Draft technical report by Robert J. Mislevy, John T. Behrens, Randy E. Bennett,
Sarah F. Demark, Dennis C. Frezzo, Roy Levy, Daniel H. Robinson, Daisy
Wise Rutstein, Valerie J. Shute, Ken Stanley, Fielding I. Winters.
Evidentiary
foundations of mixture item response theory models. By Robert Mislevy,
Roy Levy, Marc Kroopnick, and Daisy Rutstein.
A
Taxonomy of Adaptive Testing: Opportunities and Advantages of Online Assessment.
By Roy Levy, John Behrens, and Robert Mislevy.Structural,
Functional and Semiotic Symmetries in Simulation-based Games and Assessments.
By Robert Mislevy, John Behrens, Dennis Frezzo, Marc Kroopnick, and Daisy
Rutstein. Technology
Supports for Assessment Design. By Robert J. Mislevy, Isaac I. Bejar,
Randy E. Bennett,Some
Implications of Expertise Research for Educational Assessment. Keynote address
at the 34th International Association for Educational Assessment (IAEA) Conference,
Cambridge University, September 8, 2008.
How
Cognitive Science Challenges the Educational Measurement Tradition. Web
commentary for MEASUREMENT:If
Language is a Complex Adaptive System, What is Language Testing? Robert
Mislevy & Chengbin Yin. Presented at “Language as a Complex Adaptive
System,” an invited conference celebrating the 60th anniversary of
Language Learning, at the University of Michigan, Ann Arbor, MI, November
7-9, 2008. The work was supported by a grant from the Spencer Foundation.
Validity
from the Perspective of Model-Based Reasoning. Presented at the conference
“The Concept of Validity: Revisions, New Directions and Applications,”
University of Maryland, College Park, MD October 9-10, 2008. The work was
supported by a grant from the Spencer Foundation.
Design
Patterns for Learning and Assessment: Facilitating the Introduction of a Complex
Simulation-Based Learning Environment into a Community of Instructors.
Dennis Frezzo, John Behrens, and Robert Mislevy. Presented at the Annual Meeting
of the American Educational Research Association, Chicago, Il, April, 2007.
Psychometric
and Evidentiary Approaches to Simulation Assessment in Packet Tracer Software.
Dennis C. Frezzo, John T. Behrens, Robert J. Mislevy, Patti West, and Kristen
E. DiCerbo. Presented at the Fifth International Conference on Autonomic and
Autonomous Systems , April 20-25, 2009, Valencia, Spain. Best paper award.Options
for Adaptivity in ComputerAssisted Language Learning and Assessment.
Robert Mislevy, Carol A. Chapelle, Yoo-Ree Chung, Jing Xu. Presented at the
conference "Towards Adaptive CALL: Natural Language Processing for Diagnostic
Language Assessment," September 21-22, 2007, Iowa State University, Ames,
IA.
Evidence-Centered Design for Simulation-Based
Assessment.
Robert Mislevy. CRESST Report 800, July 2011.