On-line or Downloadable Papers

* ECD_overview. Brief overview of evidence-centered assessment design, from Frase, L.T., Chudorow, M., Almond, R.G., Burstein, J., Kukich, K., Mislevy, R.J., Steinberg, L.S., & Singley, K. (in press). Technology and assessment. In H.F. O'Neil & R. Perez (Eds.), Technology applications in assessment: A learning view. [html document]

* FourProcess. Almond, R.G., Steinberg, L.S., & Mislevy, R.J. (1999). A sample assessment using the four process framework. White paper prepared for the IMS Working Group on Question and Test Inter-Operability. Princeton, NJ: Educational Testing Service. [pdf format]

 

* WhereNumbersComeFrom. Mislevy, R.J., Almond, R.G., Yan, D., & Steinberg, L.S. (1999). Bayes nets in educational assessment: Where do the numbers come from? In K.B. Laskey & H.Prade (Eds.), Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence (437-446). San Francisco: Morgan Kaufmann. [preprint] [pdf format]

* ProbReasoning. Mislevy, R.J., & Gitomer, D.H. (1996). The role of probability-based inference in an intelligent tutoring system. User-Modeling and User-Adapted Interaction, 5, 253-282. [preprint] [pdf format]

* Leverage. Mislevy, R.J., Steinberg, L.S., Almond, R.G., Haertel, G., & Penuel, W. (in press). Leverage points for improving educational assessment. In B. Means & G. Haertel (Eds.), Evaluating the effects of technology in education.

* MakingSense. Mislevy, R.J., Steinberg, L.S., Breyer, F.J., Almond, R.G., & Johnson, L. (in press). Making sense of data from complex assessment. Applied Measurement in Education. [preprint] [pdf format]

* MarketBasket Paper. Prepared for NAS Conference on marketbasket reporting in Washington D.C., February 6-7, 2000. Updated version includes examples and figures. [pdf format -- 94 pages]

* Linking Educational Assessments: Concepts, Issues, Methods, and Prospects. This paper describes the basic concepts of linking educational assessments. Although it discusses statistical machinery for interpreting evidence about students' achievements, this paper's principal message is that linking tests is not just a technical problem. Technical questions can not be asked or answered until questions about the nature of learning and achievement and about the purposes and consequences of assessment are addressed. Some fundamental ideas about educational assessment and test theory (chains of inferences and roles of judgment) are considered. The following approaches to linking assessments are described: (1) equating in physical measurement and educational assessment; (2) calibration in physical measurement and educational assessment; (3) projection in physical measurement and educational assessment; and (4) statistical moderation and social moderation. Implications for a system of monitoring progress toward educational standards are discussed. 1992. [pdf format -- about 90 pages]

* Linking Adult Literacy Tests. 1995 paper commissioned by the Department of Labor and the Department of Education, on prospects, methods, and issues on the topic of linking results across the several literacy assessments that are used to evaluate adult literacy programs. [pdf format -- about 60 pages]

* Conditional Probability Models. "Models for condtional probability tables in educational assessment," by R.G. Almond, L. Dibello, F. Jenkins, R.J. Mislevy, L.S. Steinberg, and D. Yan. Paper presented at the 2001 AI in Statistics conference. Concerns developing Bayesian priors based on subject matter experts' knowledge about the structure and the difficulty of complex assessment tasks. [pdf format -- 7 pages]

* Psychometric Principles. “Psychometric Principles in Student Evaluation,” by R.J. Mislevy, M.R. Wilson, K. Ercikan, & N. Chudowsky. 9/30/01 draft of a chapter to appear in D. Stufflebeam & T. Kellaghan (Eds.), International Handbook of Educational Evaluation. Dordrecht, the Netherlands: Kluwer Academic Press. [pdf format – 68 pages]

* Modeling conditional probabilities. “Modeling conditional probabilities in complex educational assessments,” by R.J. Mislevy, D. Senturk, R.G. Almond, L.V. Dibello, F. Jenkins, L.S. Steinberg, & D. Yan. Paper presented at the International Meeting of the Psychometric Society, July 15-19, 2001, Osaka, Japan. "Effective theta" method for expressing experts' knowledge in Bayesian prior distributions, MCMC estimation to refine parameters, and illustrations from the Biomass project. Extensions of Almond et al. AIStat 2001 paper. [pdf format -- 56 pages]

* Task-Based Language Assessment. "Design and analysis in task-based language assessment," by Robert J. Mislevy, Linda S. Steinberg, and Russell G. Almond. To appear in a special issue of Language Assessment, John Norris, guest editor. [pdf format – 32 pages]

* Metropolis sampling. "Metropolis Sampling for a 2-Valued Random Variable." Heuristic arguments for why Metropolis sampling works, and the rationale for "burn-in" cycles. Revised 12/8/01. [pdf format -- 5 pages]

* Structure of Assessments. Preprint draft of "On the structure of educational assessments," by Robert J. Mislevy, Linda S. Steinberg, and Russell G. Almond. To appear in Measurement. [pdf format – 84 pages]

* Biomass Overview. The Biomass project at ETS produced a running prototype of a web-delivered, standards-based assessment of science inquiry skills. It demonstrates the use of an evidence-centered approach to assessment design, and the implementation of a four process delivery system. [pdf format – 37 pages]

* Delivery Architecture & Assessment Design. Almond, R.G., Steinberg, L.S., & Mislevy, R.J. (2002). "A Four-Process Architecture for Assessment Delivery, with Connections to Assessment Design. " To appear in the Journal of Learning Technology and Assessment. Lays out connections between the models of the ECD assessment design framework and the processes of the four-process delivery system architecture. [pdf format -- 67 pages]

* Argument Substance and Argument Structure in Educational Assessment. To appear in Law, Probability, and Risk. Presented at the Conference on Inference, Culture, and Ordinary Thinking in Dispute Resolution, Benjamin N. Cardozo School of Law, Yeshiva University, New York, New York, April 27-29, 2003. [pdf format -- 39 pages]

* Intuitive Test Theory. Draft of "Intuitive Test Theory," by Robert J. Mislevy & Henry I. Braun. This paper is the text of an invited address by the first author at the Annual Dinner Meeting of the Princeton Association for Computing Machinery (ACM) and Institute of Electrical and Electronics Engineers (IEEE) Computer Society Chapters, in Kingston, NJ, May 22, 2003.

* Brief Intro to ECD. By Robert J. Mislevy, Russell G. Almond, and Janice Lukas. A brief introduction to "evidence centered assessment design" (ECD). Includes a glossary and links to web material for further reading.

* Correspondence concerning normalized gain scores. Two pieces of correspondence motivated by a posting by Prof. Richard Hake on the AERA-D listserv at <http://lists.asu.edu/cgi-bin/wa?A2=ind0603&L=aera-d&T=0&F=&S=&P=339> concerning the use of normalized gain scores <g> to measure learning. 

*Implications of Evidence-Centered Design for Educational Testing. Preprint draft of "On the structure of educational assessments," by Robert J. Mislevy & Geneva D. Haertel. To appear in Educational M easurement: Issues and Practice.
* On the Roles of External Knowledge Representations in Assessment Design. Draft technical report by Robert J. Mislevy, John T. Behrens, Randy E. Bennett, Sarah F. Demark, Dennis C. Frezzo, Roy Levy, Daniel H.  Robinson, Daisy Wise Rutstein, Valerie J. Shute, Ken Stanley, Fielding I. Winters.
*Evidentiary foundations of mixture item response theory models. By Robert Mislevy, Roy Levy, Marc Kroopnick, and Daisy Rutstein.
 
*A Taxonomy of Adaptive Testing: Opportunities and Advantages of Online Assessment. By Roy Levy, John Behrens, and Robert Mislevy.
 
*Structural, Functional and Semiotic Symmetries in Simulation-based Games and Assessments. By Robert Mislevy, John Behrens, Dennis Frezzo, Marc Kroopnick, and Daisy Rutstein.
 
*Technology Supports for Assessment Design. By Robert J. Mislevy, Isaac I. Bejar, Randy E. Bennett,
Geneva D. Haertel, & Fielding I. Winters.
*Some Implications of Expertise Research for Educational Assessment. Keynote address at the 34th International Association for Educational Assessment (IAEA) Conference, Cambridge University, September 8, 2008.
 
*How Cognitive Science Challenges the Educational Measurement Tradition. Web commentary for MEASUREMENT:
Interdisciplinary Research and Perspectives
, Volume 6, Numbers 1 & 2.
*If Language is a Complex Adaptive System, What is Language Testing? Robert Mislevy & Chengbin Yin. Presented at “Language as a Complex Adaptive System,” an invited conference celebrating the 60th anniversary of Language Learning, at the University of Michigan, Ann Arbor, MI, November 7-9, 2008. The work was supported by a grant from the Spencer Foundation.
 
*Validity from the Perspective of Model-Based Reasoning. Presented at the conference “The Concept of Validity: Revisions, New Directions and Applications,” University of Maryland, College Park, MD October 9-10, 2008. The work was supported by a grant from the Spencer Foundation.

*Design Patterns for Learning and Assessment: Facilitating the Introduction of a Complex Simulation-Based Learning Environment into a Community of Instructors. Dennis Frezzo, John Behrens, and Robert Mislevy. Presented at the Annual Meeting of the American Educational Research Association, Chicago, Il, April, 2007.

*Psychometric and Evidentiary Approaches to Simulation Assessment in Packet Tracer Software. Dennis C. Frezzo, John T. Behrens, Robert J. Mislevy, Patti West, and Kristen E. DiCerbo. Presented at the Fifth International Conference on Autonomic and Autonomous Systems , April 20-25, 2009, Valencia, Spain. Best paper award.
 
*Options for Adaptivity in ComputerAssisted Language Learning and Assessment. Robert Mislevy, Carol A. Chapelle, Yoo-Ree Chung, Jing Xu. Presented at the conference "Towards Adaptive CALL: Natural Language Processing for Diagnostic Language Assessment," September 21-22, 2007, Iowa State University, Ames, IA.

*Evidence-Centered Design for Simulation-Based Assessment. Robert Mislevy. CRESST Report 800, July 2011.