May 10, 2007

Correction: Project is due May 18.

May 3, 2007

May 7 is the last class meeting for the semester.  Here are plans for the meeting:

First half of class: ½ hour group work to prepare for discussion on task models reverse-engineered from the WebLAS examples, then full class discussion.

Questions/discussion about final project (which is due December 19).

Retrospective on the class: For next time I teach this class...

• What was useful, what wasn't?
• Group projects rather than individual ones?
• What to emphasize more, what to de-emphasize?
• Which readings were useful, which weren't?
• Discuss readings explicitly in class?
• Have students present readings?
• Different homework, more homework, less homework; add tests?

April 24, 2007

In the April 30 class meeting, we will have another in-class exercise sketching task models. You may stay in the same groups or form new ones, each with 2-4 members.

We will have a common starting point this time: Korean Listening tasks from the WebLAS project. The WebLAS project at UCLA represents collaboration efforts of the Principal Investigator, Dr. Lyle Bachman, the Language Subject-Matter Experts, and the WebLAS Software Development Team in a joint conjunction between Applied Linguistics and the Center for Digital Humanities. I would like you to go to the homepage, and explore the sample materials for the Korean Placement Examination. Check out the design statement, the task specifications, and the tasks of Sample Listening Test 1. (Warning--these tasks worked well on some computers I tested them on, but had troubles on some other computers).

We will again focus on the following attributes:

• Brief description of the task model
• Sketch of warrant
• Characteristic task features
• Task model variables
• Stimulus materials, if any
• Conditions/tools/instructions for performance
• Work product(s)

You will find that the WebLAS team has done quite a bit of thinking about the frameworks under which they have created their tasks, exploiting ideas we have discussed in connection with ECD.

You can define a task model at a more general level that would encompass all three of the sample tasks and provide a framework others for like them at this general level, or at a more specific level by reverse-engineering from one of the tasks to provide a framework for new tasks in a more closely related family that more strongly resemble the one you worked back from.

At 6:30, we will do the course evaluation forms.

April 17, 2007

In the April 23 class meeting, we will have an in-class exercise sketching task models from the topics we used March 12 for exploring evaluation procedures. Each group will outline a task model for their topic that addresses the following attributes:

• Brief description of the task model
• Sketch of warrant
• Characteristic task features
• Task model variables
• Stimulus materials, if any
• Conditions/tools/instructions for performance
• Work product(s)

Last class meeting I mentioned a chapter from the forthcoming volume from the "Idea of Testing" book, discussing some aspects of assessment arguments and procedures connecting with a situative/sociocultural perspective on learning. It is entitled "Issues of Structure and Issues of Scale in Assessment From a Situative/Sociocultural Perspective." There is a CRESST Research Report version available at http://www.cse.ucla.edu/products/reports/r668.pdf.

April 4, 2007

April 9: No class meeting.

April 16: Starting on new topic, Task Models and Theory-Based Task Design. We will begin to work through four sets of overheads on task design

April 1, 2007

Recall that we will use MSBNx for small-group in-class exercises on April 2 concerning these problems. Note that although the page they are on says "Assignment 5," this is not an assignment to be handed in. Marc Kroopnick will be our guest instructor for the first half of the class. In the second half of the class, we will continue working on these problems if that is warranted, or discuss some more Bayes nets.

The National Center for Research on Evaluation, Standards, Student Testing (CRESST), Center for Studies in Education, UCLA has updated their website. The good news is that it looks great. The bad news is that the previous links to the research reports don't work any more. I have fixed them on the Readings page. Alternatively, you can go to www.cse.ucla.ed, click on the Reports link near the top of the page, and enter the paper number to get a paper. (Hat tip to Joyce)

We will next address the measurement model component of the Evidence Model. We will begin on March 26 with the presentation Introduction to Bayes nets.

Download the free computer program MSBNx, the Microsoft Bayes net editor from the Microsoft website. It just runs on Windows machines--sorry Mac users! We will talk about this program a bit on March 26, and build a simple model or two in class together. Bring your computer with MSBNx if you have it, so you can get the problem running on your own computer. The reading "Evidence and inference in educational assessment" will give you a pretty good idea of what Bayes nets are about, and if you are interested in digging deeper, the Help drop-down menu in MSBNx includes a tutorial on how to set up and run problems.

We will use MSBNx for small-group in-class exercises on April 2 concerning these problems. Marc Kroopnick will be our guest instructor for the first half of the class, and will help you as you do this. Definitely have MSBNx up and running on your portable computer and bring it to class that day.

Another potential topic to choose in the in-class discussion of evluation procedures: a televison show such as Iron Chef, Project Runway, Top Chef, etc., in which contestants are assigned a task in terms of goals and constraints which has supposedly been designed to provide evidence about some salient aspects of the proficiency of interest. If you do this one, use a specific challenge to discuss evaluation procedure for.

An example of a knowledge representation used to support distributed cognition at the waffle house.

March 6, 2007

The optional reading on psychological perspectives I mentioned in class is Greeno, J.G., Collins, A.M., & Resnick, L.B. (1997).  Cognition and learning.  In D. Berliner & R. Calfee (Eds.), Handbook of educational psychology (pp. 15-47).  New York: Simon & Schuster Macmillan. Turns out there is not an online version from the library. I will see if I can find a paper copy for you to borrow.

Plans for March 12 class meeting: In-class exercise concerning evidence evaluation. We will break into 3 groups. Each group will choose one of the topics below to analyze, in terms of the following issues:

• What is the nature of the proficiency or proficiencies of interest?
• What is the rationale for the setting in which performance is observed?
• What is the work product?
• What aspects of the performance, as expressed in the work product, are identified and evaluated?
• Why are these aspects believed to provide information about the proficiency(ies)?
• Are the criteria for evaluation explicit or implicit? How are they applied?
• Are there demands for construct irrelevant proficiency involved in the performance?

You will discuss and analyze them before the break, and report to the class your conclusions after the break.

The topics are as follows:

• Evaluation of performance in science tasks, in the Baxter, Elder, & Glaser reading.
• Evaluation of performance in simulated patient management problems, in the Clauser et al. (1997) reading.
• Performance test to determine awarding a particular belt in some martial art (e.g., what is the assessment, and how is it evaluated, to move from a white belt to a yellow belt in some particular style of karate?).
• The driving part of the test for getting a drivers license.
• Evaluation of diving performances in the olympics.

By lottery, the groups will be assigned first, second, and third choice to work on. Everyone will have read the articles for the first two topics. The other three should be more familiar general knowledge, but if you have your computer you can get more information online as you work.

Plans for March 5 class meeting:

• Leverage Points for Cognitive Psychology in Assessment
• From students who have not done so, short update from them on their intended assessment example.
• A talk on Evidence Evaluation, i.e., task-level scoring

Supplementary readings: Chapters from Automated Scoring of Complex Tasks in Computer Based Testing (published by Erlbaum)edited by David Williamson, Isaac Bejar, and myself:

• Introduction
• Human scoring (pp. 1-13)
• Automated scoring of text

These are password protected files, with a password sent to registered students in an email. Let me know if you didn't receive it.

Additional short reading assignment: Section 2.1 (pp. 7-9 and Figures 2 & 3) of Cognitive Psychology and Educational Assessment (password protested--select for 'read only' if you get a second dialogue box). Figure 3 is an extended version of the Toulmin assessment-argument diagram.

Plans for Feb 26 class meeting:

• Psychology in assessment arguments
• Leverage Points for Cognitive Psychology in Assessment
• Look at ECD Wiki if UM internet is available.
• From students who were not present last week, short update from them on their intended assessment example.

Reminder: Assignment 2 on your class example is due Friday Feb 23 (unless you were taking EDMS PhD comps).

Excellent chapter on automated scoring of essays we noted in class (password protected).

There is a new version of the overheads for Expertise Research and Assessment ("director's cut" -- seven new slides).

Here is an ECD Wiki orginated by Dr. Russell Almond: http://ecd.ralmond.net/ecdwiki/pmwiki.php. We will talk about it in class. By a separate email I am forwarding to you information from Dr. Almond about what it is, how to access it, and how to contribute to it. (Let me know if you don't get the email--the class list of emails I have may not be complete.)

On Monday, February 19, we will continue discussing evidentiary reasoning and assessment, with evidentiary reasoning in assessment. We will spend a litle time with your brief introductions to assessments you are thinking about for your class project.

The next topic will be psychology and assessment. The overheads for that are now up:

• Psychology in assessment arguments
• Leverage Points for Cognitive Psychology in Assessment
• Expertise Research and Assessment

Assignment 1, ECD, is due next Friday, February 16.

In the last class meeting, we talked about the start-up costs of learning new, potentially better, ways to do things--using ECD forms, for example, or switching typewriter keyboard layouts after one was already proficient with an alternative. The standard typewriter/computer keyboard layout is QWERTY. The most common alternative, the name of which I couldn't recall, is Dvorak.

February 7, 2007

On Monday, February 12, we will finish up a few points from an introduction to the four-process delivery system. We will then start to discuss evidentiary reasoning and assessment. There are two sets of overheads, concerning an introduction to evidentiary reasoning and evidentiary reasoning in assessment.

Assignment 1, ECD, is due next Friday, February 16. Be prepared to speak informally in class on February 19, to tell the rest of the class a little about your example. No need to prepare overheads or handouts.

The "tell me about yourself" assignments are beginning to roll in--thanks!

On Monday, February 5, we will start with a look at the layers in assessment design. We will discuss ECD issues from the ECD readings--your questions, your thoughts so far, and the following seed question:

• What are advantages and disadvantages of having a formal framework for assessment design? (i.e., is all this work really worth it?)

We will then start the presentations on the four-process delivery system architecture. There are overheads for an introduction to the four-process delivery system and examples of the four-process delivery system.

Welcome to EDMS 738 for Spring 2007, "Foundations of Assessment"! Our class will meet Mondays 4:15-6:50 Eastern time, in the Engineering Annex Building 0307.

The first meeting is January 29. The first half of the class will be introductions and an overview of the course. After a break will be an initial presentation on the basic models of the evidence-centered design framework, which we will be using throughout the course to organize our thinking. If you are reading this note before our first class meeting, read this research report from our reading list before class: A brief introduction to evidence-centered design, by Mislevy, Almond,& Lukas.

On the class website you will find a link to overheads for the class, which now contains overheads for the first class meeting, Course Overview and Intro to Evidence-Centered Assessment Design. Once we are up and running I will aim to get overheads onto the website by end of day Friday before a class meeting. This will allow you to download and print copies of overheads if you wish, so you have copies you can make notes on during presentations and discussions. Overheads that are powerpoint slides such as these, you may want to download to your computer and print them out 2, 3, or 6 per page to save paper.

Feel free to contact me if you have any questions. My email address is rmislevy@umd.edu.

Bob Mislevy