EDMS 738: Foundations of Assessment

Spring 2007

Prof. Robert Mislevy

1230-C Benjamin Building

rmislevy@umd.edu

Course Description: This course is built around a model for evidence-centered assessment design. It introduces the student, evidence, and task models of the conceptual assessment framework, and explores their roles in a four-process delivery system. This perspective applies to assessments that range from large-scale standardized tests and classroom quizzes, to intelligent tutoring systems and sessions between students and tutors. The course therefore emphasizes the principles of evidentiary reasoning and design-under-constraints that are common across assessment forms and purposes. After an overview of the models, the class explores how a coherent assessment design builds on each of several disciplines, including cognitive and situative psychology, evidentiary reasoning, measurement models, and Bayesian statistical inference. Ideas will be illustrated with examples including the Advanced Placement Studio Art Portfolio Assessment, the GRE, and the Hydrive coached practice system. As a course project, each student will explicate an assessment (preferably ones in which they have a professional interest) in terms of the design and delivery-system models.

Prerequisites: EDMS 623 (intermediate-level measurement—e.g., test development, validation) and EDMS 651(Intermediate-level statistics— e.g., regression, anova), or equivalent.

Course materials: The readings for this course are a set of research reports, journal articles, and chapters from edited volumes. All are available online, either as links to public versions or password protected files for registered students only.

Course calendar: Weekly calendar with topics, readings, and assignments.

Course Evaluation:

There are six assignments: a question set, five short papers, and a term paper. Their weights in your grade are 11% for each short assignment and 33% for the term paper.

All assignments will be submitted by email, as a Word document.

Unless otherwise stated, assignments are due by midnight Friday of the week they appear in the course calendar.

An additional ungraded start-up assignment is a paragraph about yourself. The objective is for me to get to know you so I can aim assignments, talks, and discussions appropriately. Tell me a bit about your educational and work background, including measurement and statistics, and experiences with assessment.

The stand-alone assignment is a question set about evidence centered design based on the readings.

The five short papers are two-to-four page papers in which you relate the ideas in the readings and class discussions to an assessment of your choosing. The first is a description of your assessment: how it is used, what it looks like, whether you have personal experience or interest in it, etc. The subsequent topics are the psychological perspective of your assessment, task scoring, statistical model, and task design rationale. In each case we will discuss the concepts in class.

An existing assessment you are already familiar with is probably the best choice, but it would also work to choose one you want to learn about or are in the process of creating. You can switch examples along the way if your initial choice is not working for some reason—talk with me if you see this happening. You may work in pairs or groups if you like, on a shared example. However, each student in a group must write his or her own papers.

The term paper will discuss your class example in greater detail, addressing each of the topics we have discussed: Student-, evidence-, and task-models; psychological perspective; statistical model; task-level scoring; and delivery cycle. We will have had an opportunity to discuss these topics in class, and your short papers will give you a great start. The final project must be turned in by May 18, Friday, of the final exam week. The term paper should be 15-25 typed, double-spaced, pages. It does not count for your grade, but it provides an opportunity for us to talk about what you plan to do, to make sure you are on track.

Grading: Assignments will be graded on a 0 to 4 point scale where:

4=Outstanding (Good effort, on target, and complete performance)

3=Good (Good effort shown in work. Only minor problems in performance)

2=Acceptable (The performance is moderately flawed, but is acceptable.)

1=Unacceptable (The performance is not indicative of graduate level work, is severely flawed or is indicative on a substandard level of effort.)

0=Assignments that are not turned in or are indicative of such bad performance that they should not have been turned in.

Before each assignment, I will tell you what I need to see to attain a 4. Generally, it requires showing how certain concepts we have read about and discussed in class apply to the assessment you have chosen as your example.

Final Grades will be calculated by weighting each individual assignment grade according to the weights defined in the previous section. Numeric grades will be translated as follows:

A= 3.50001 - 4.0

B= 2.75001 - 3.5

C= 1.75001 - 2.75

D= 0.75001 - 1.75

F = 0.00000 - 0.75

Please note that there will be no opportunity for “extra credit”. Grades will be determined by the scheme outlined above - no exceptions.

Late Assignments: Homework should be submitted by midnight Friday of the week it is due unless special arrangements are made with me in advance. Moreover, special arrangements will only be instituted under extenuating circumstances, and thus, assignments should generally be turned in on time. If an assignment is turned in late without advanced approval, then the grade received for the assignment will be automatically decreased by 1 point for each week or portion thereof it is late.

Grades of “Incomplete”: A grade of incomplete will generally not be allowed except in cases of extreme hardship.

Honor System: Each student is expected to complete all assignments independently except as otherwise allowed--e.g., with my prior approval, working in groups on papers or projects. There will be plenty of opportunity in class to discuss your projects and assignments, and these discussions may continue outside class. However, the write-up must be your own work, and you are expected to show that you understand it in your in-class summary of the assignment. If you need help with some aspect of your assignment, then come to me and I will assist you to a reasonable extent.

Accommodations: If you need academic accommodation by virtue of a documented disability, please contact me as soon as possible to discuss your needs. Students with documented needs for such accommodation must meet the same achievement standards required of all other students, although the exact way in which achievement is demonstrated may be altered. If you would like academic accommodation by virtue of your religion (e.g., turning in homework at a time other than the due date because that date falls on a religious holiday), then please contact me as soon as possible to discuss your request. All requests for academic accommodations should be made within two weeks of the start of class.

Auditors: For individuals who are auditing the course can attend class as much or little as they would like, reading the assigned material and performing the homework problems is encouraged, but not required.