What You Test is What You Get

Q&A with Robert J. Mislevy, Ph.D., whose research is on the cutting edge
of assessing what students know

Robert J. Mislevy is a professor in the College of Education’s Department of Measurement, Statistics, and Evaluation. He is an internationally-known scholar in cognitive research who was elected in May 2007 to membership in the National Academy of Education. Mislevy’s current research focuses on developing new methods of “open-ended assessment,” a type of testing that offers advantages over established multiple-choice and short-answer testing. Prior to joining the University of Maryland in 2001 Mislevy spent 16 years at Educational Testing Service.

Q: What is open-ended assessment, and how does it differ from short-answer or multiple choice assessment?
A: Open-ended assessment is testing in which a person has to actually construct a response, such as solving an architectural design problem or troubleshooting a broken computer network or writing an essay. It gives individuals the opportunity to construct solutions and show evidence of their understanding of a process. Closed assessments, such as short-answer or multiple-choice tests, only allow persons to choose within a small, narrowly-defined solution space. They yield only bits of information about what the person knows.

Q: "Standards" and "accountability" have become national buzzwords, especially since No Child Left Behind. However, in your research you found the short answer assessments that are now in vogue do not really serve students or schools well. How so?
A: Although closed assessments provide some information, they are not very good at some very important things. For example, at one point Cisco Systems—a global computer networking and communications company—was using multiple-choice and short answer tests to evaluate students in its computer networking classes worldwide. The trouble was that a student could do well on those tests, but then sit down in front of a broken computer network and have no idea where to get started. I am now working with the Cisco team on open-ended assessments in which, in order to do well on the test, students must actually know how to do computer troubleshooting. This goes beyond the merely declarative knowledge that was being tested before. Open-ended assessment measures a broader range of what we want students to learn. It can be visual, it can be analytical, or it can be a sequence of actions. My research asks: How do we create sophisticated open-ended testing that gives schools and employers affordable access to better measurement of knowledge?

Q: Is moving toward open-ended assessment a controversial idea in the field?
A: Some still embrace a false dichotomy between "objective" closed assessments and "squishy" open-ended assessments. The research that I and others are doing is using principles of evidentiary reasoning, statistical models, and computer-automated scoring to develop open-ended testing that is scientifically sound. Once we can do that, the argument is over. I think most scholars in this field, if they were to look closely at our work, would approve of it.

Q: How far along are you and your colleagues in developing effective open-ended assessments for wide-scale use?
A: Our work has come into use in bits and pieces during the past decade. Through a University of Maryland program called Principled Assessment Design for Inquiry in Science (PADI), I’ve been working with a University of Michigan project called BioKids. For this, we are creating open-ended assessments that teachers in the Detroit Public Schools can use to measure the performance of students who have gone out and studied insects and biodiversity in the city. I’m also part of a proposal to bring more open-ended testing into the Minnesota State Proficiency Test. Other states are moving in this direction, including Maryland. We’ve made great progress in understanding how to do open-ended testing. The next big step is in making it easily accessible for educators. I would say we’re roughly halfway there. Once we’re able to offer software tools and templates that can be cheaply and easily used and reproduced, then you’ll see things ramp up quickly.

Q: How big a challenge will it be to make open-ended assessment affordable for K-12?
A: It used to be that open-ended testing required you to have an expert evaluate every student’s responses. That was very expensive. Now we have the technologies to accurately carry out automated scoring of open-ended assessments. The testing we are developing for Cisco Systems, for instance, can be scored automatically with software. As in any field, the upfront research is expensive, but once you’ve done it, people will have access to a family of affordable testing resources.

Q: Do you believe that open-ended assessment will ultimately become the norm nationally?
If so, how soon?
A: Traditional forms of assessment are still very good at what they do. I simply see a need to expand the array of testing tools. I think states such as Maryland and Minnesota are aware of the limitations of closed assessments, but they’re facing federal mandates and limited budgets. I see more and more states expanding their testing methods as open-ended testing becomes more accessible. I think it’s reasonable to predict that within the next five to 10 years states will be doing their testing with a 50/50 ratio of open-ended versus closed assessments. Accountability is a good thing. However, I think it’s a false equation to identify accountability solely with single-event, multiple-choice or short-answer tests. The strength of open-ended assessment is that if we do it right, we are testing students’ ability to think about the real world: the principles of physics, the principles of data analysis, the processes of history, and so on. The basic idea is, better assessment leads to better learning. As researcher Lauren Resnick of the University of Pittsburgh has said, "What you test is what you get."

(Proessor. Mislevy was interviewed by Bruce Jacobs for the September 2007 issue of Endeavors— a publication for alumni and friends of the College of Education)