Last night was the final class meeting. It has been an enjoyable class for me--good discussions, interesting projects, pleasant company.

Two of the journal articles we discussed are now available online, both password protected to restrict access to UM registered students in the class:

• Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13-23.
• White, B.Y., & Frederiksen, J.R. (1998). Inquiry, modeling, and metacognition: Making science accessible to all students. Cognition and Instruction, 16, 3-118.

Also up on the website is the description of the Final Project assignment. It is due December 19.

Plans for final class meeting on December 12:

• Questions/discussion about final project (which is due December 19).
  
  
• Open discussion (with your observations and questions from two weeks ago as seeds)
  
  
• Retrospective on the class: For next time I teach this class...
  
  
  
  • What was useful, what wasn't,
  • What to emphasize more, what to de-emphasize,
  • Which readings were useful, which weren't,
  • Different homework, more homework, less homework; add tests?

Yesterday we talked a bit about the contributions of cognitive research on reading to reading instruction. As a supplemental reading in case you are interested in learning more about this, a very good review article about this topic is here.

Plans for the December 5 meeting:

• Presentation based on Embretson reading
  
• In-class group work and discussion based on the Web-based Language Assessment System (WebLAS) project at UCLA. The material here represent collaboration efforts of the Principal Investigator, Dr. Lyle Bachman, the Language Subject-Matter Experts, and the WebLAS Software Development Team in a joint conjunction between Applied Linguistics and the Center for Digital Humanities.

Assignment 5 is now up. It is due Friday December 9.

Hope you all had a good holiday. On Monday November 28, we will continue the discussion of task models, beginning with the presentation on Hively et al. and then the one on task model variables. In the second half of the class we will lay out the elements of a task model together, reverse engineering from a task that appeared on an EDMS comprehensive assessment.

Following up on a couple of points that came up in our discussion last Monday:

• A chart showing the correspondence between attributes of design patterns and the basic ECD models.
  
  
• Susan Embretson refers to the two angles of validity argumentation as nomothetic span (correlation of test scores with other test tests scores or variables) and construct representation (building features into tasks that your theory says will call upon the targeted proficiencies). If you want further information on this, you can read more in CRESST Technical Report 583, "Psychometric principles in student assessment,", page 16, or go to following definitive article:

  Embretson, S. (1983). Construct validity: Construct representation versus
  nomothetic span. Psychological Bulletin, 93, 179-197.

• A design pattern for assessing "building scientific explanations" created by the BioKIDS project at the University of Michigan. Here is an example of a task motivated by this design pattern (without its accompanying diagram):

  "If all of the small fish in the pond system died one year from a disease that killed only the small fish, what would happen to the algae in the pond? Explain why you think so. "

No class meeting November 14; class meets next November 21. I am looking forward to seeing you again! Many thanks to Roy, Michelle, and Geneva for filling in for me.

We will begin to talk about task design and task models. We will begin to work through four sets of overheads on task design starting November 21.

For the following weeks, half of the class will be discussion and the other half lecture.

For the November 21 discussion: Please prepare and bring to bring to class on note cards two things, based on your continued reading, class meetings, and thinking about your projects from the perspective of evidence-centered assessment design:

• One observation. This could be an insight about assessment, a point visited earlier in the class that makes sense to you in a different way now, a principle you have realized about your own project, or a quote or idea from one of the readings or elsewhere that captures an important idea about assessment. It could be simple or profound, a detail or big picture--just something that is significant to you.



• One question. Again, big or small, simple or profound, based on readings, presentations, projects, or something related to assessment in the world; maybe there's an answer, maybe there isn't but it just gets us thinking.

Bayes nets will be a topic addressed in Roy Levy's guest lecture on October 24. The "Evidence and inference" reading assigned for that day will give you the background you need for the talk. Optionally, you can download and build networks using Microsoft's free Bayes net editor MSBNx.

Assignment 3, regarding psychology and your example, is due October 21. There are examples of papers from past classes for this assignment available for your perusal.

Assignment 4, regarding evidence evaluation and your example, is due November 11. There are no examples from past classes for this assignment but you know the game by now. And next time I teach the class, yours can be up as examples for your intellectual heirs (with your permission).

We will not have an assignment on statistical models and your assessment.

Now up: Overheads for Roy Levy's guest presentation on Bayes nets in assessment, October 24.

On Monday, October 17, we will meet in our new location, Benjamin 0306--Starting, I hope, at 4:15.

We will have a short talk on psychology and assessment, using Leverage Points for Cognitive Psychology in Assessment, and a talk on Evidence Evaluation, i.e., task-level scoring.

Supplementary readings are chapters from a volume in press edited by David Williamson, Isaac Bejar, and myself, entitled Automated Scoring of Complex Tasks in Computer Based Testing (Lawrence Erlbaum publisher):

• Introduction
• Human scoring (pp. 1-13)
• Automated scoring of text (Dongyang especially!)

These are password protected (with a password you received in an email). Please do not cite, quote, or share this material.

On Monday, October 10, we will meet in our new location, Benjamin 0306.

We will address psychology and assessment, using the following overheads:

• Psychology in assessment arguments
• Leverage Points for Cognitive Psychology in Assessment
• Expertise Research and Assessment

Supplementary reading now up: My recently completed chapter entitled Cognitive psychology and educational assessment (password protected), for the upcoming Educational Measurement, Fourth Edition, edited by Robert Brennan.

• Section 1: Introduction
• Section 2: Assessment as evidentiary argument, plus a first look at probability models
• Section 3: Cog psych themes and implications for assessment
• Section 4: Measurement models for cognitive diagnosis
• Section 5: Conclusion

Sections 1-3 are particularly relevant for things we're discussing in class currently. Section 4.1 will be relevant for our consideration of probability-based reasoning, in a couple weeks from now. Section 4.2 goes beyond what we're doing this semester, but you may find it of interest.

Up on the website: Assignment 2, ECD questions. Due Friday October 7.

On Monday, October 3, we will continue discussing evidentiary reasoning and assessment, with evidentiary reasoning in assessment. The second half of class will be your brief introductions to assessments you are thinking about for your class project.

The next topic will be psychology and assessment. The overheads for that are now up:

• Psychology in assessment arguments
• Leverage Points for Cognitive Psychology in Assessment
• Expertise Research and Assessment

Our meeting will be in the same classroom, ENG 3103, on Monday. However we will move to a different classroom the following week, and thereafter. I will tell you in class (and post it on this class news page) when the arrangements are finalized.

My collaborator Dr. Russell Almond, Senior Research Scientist at ETS, is visiting the University of Maryland on Monday October 3 also. He will present a talk on evidence-centered assessment design (abstract) at EDMS's Measurement & Statistics Monday Symposia, 11:00-12:00, in Benjamin 1121.

I have updated the class calendar to reflect our guest speakers:

On Monday, September 26, we will discuss evidentiary reasoning and assessment. There are two sets of overheads, concerning an introduction to evidentiary reasoning and evidentiary reasoning in assessment. I hope to get through both of these in class, and if not we will finish them up next week.

Assignment 1, your example assessment, is due next Friday, September 30. Be prepared to speak informally in class on October 3, to tell the rest of the class a little about your example. You don't need to prepare overheads, but you may if you wish, with an upper limit of 3.

September 16, 2005

The "tell me about yourself" assignments are beginning to roll in--thanks!

On Monday, September 19, we will start with a brief look at the layers in assessment design. We will discuss ECD issues from the ECD readings--your questions, and the following seed questions:

• The main kinds of variables in the ECD framework are student model variables, observable variables, and task-model variables. What are they conceived as characteristics of? How are they related to each other? Do they exist independently of one another?
• What are advantages and disadvantages of having a formal framework for assessment design? (i.e., is all this work really worth it?)

We will than start the presentations on the four-process delivery system architecture. There are overheads for an introduction to the four-process delivery system and examples of the four-process delivery system.

Now available on the class website:

• Syllabus
• Assignments page.

September 13, 2005

It was a pleasure to meet you all last night! I have added to the readings list the research report on AP Studio Art. We will talk about it when we discuss evidence evaluation and performance assessment.

September 11, 2005

Course location: Engineering (Martin Building) 3103

Now available on the class website:

• Overheads
• Class calendar

Plans for September 12 class (links to the overheads--download them ahead of class and print them if you want to):

• Overview of the course
• Introduction to evidence-centered assessment design

August 22, 2005

The class reading list is now available. Note that all the readings are available online. Most are available to the general public, but a few are available to registered EDMS 738 students only.

August 13, 2005

Welcome to EDMS 738 for Fall 2005, "Foundations of Assessment"!

Our first meeting is Monday, September 12, in a location to be determined. When I know the classroom, I will post it here in Class News.

The first half of the class will be introductions and an overview of the course. After the break will be an initial presentation on the basic models of the evidence-centered design framework, which we will be using throughout the course to organize our work.

If you are reading this note before our first class meeting, read this research report from our reading list before class: A brief introduction to evidence-centered design, by Mislevy, Almond, & Lukas.

Bob Mislevy