EDMS 738: Theory-Based Task Design

 

Syllabus

 

Spring 2004

Prof. Robert Mislevy

1230-C Benjamin Building

rmislevy@umd.edu

 

 

Course Description: This course explores theory-based design of educational assessment tasks.  We will use the framework of ‘evidence-centered assessment design’ (ECD) to analyze a number of examples, map them into structures for assessment arguments, and create our own tasks.  The first third of the course will be an introduction to the ECD and related structures, including Toulmin diagrams for arguments and “design patterns” for recurring genres of tasks, which were developed under the NSF-supported project "Principled Assessment Design for Inquiry" (PADI).  The remaining two thirds of the class will be devoted to working through a number of examples.

This is a hands-on class.  With the introductory examples, we will work through parts of the design structures in class and students will do additional details as homework.  We will move to more complex examples, first addressing existing examples projects that have been carried out at ETS and elsewhere, then moving to examples the students select for themselves.

 

Prerequisites: EDMS 623 (intermediate-level measurement, including reliability, validity, and test development), or instructor permission.

 

Course Evaluation:

There are seven assignments:

  • An ungraded ‘start-up assignment’. The ungraded start-up assignment is a paragraph about yourself.  The objective is for me to get to know you so I can aim assignments, talks, and discussions appropriately.  Tell me a bit about your educational and work background, including measurement and statistics, and experiences with assessment.
  • Four short assignments or question sets.  Based on readings or continuations of work done in class.  Examples are creating a Toulmin diagram or design pattern for one of the examples we have addressed in common.
  • A presentation of an example from the textbook (or similar source).  After we have done some examples in class, every student will chose a chapter from the textbook (or with my permission, an alternative research article), to summarize for the class and present a design patterns and elements of a task model.
  • A term paper.  Based on assessment, research paper, or project of the student’s own choosing: Provide brief background of assessment application, explanation of assessment argument, design pattern(s), elements of task model. You may work in pairs or groups if you like, on a shared example.  However, each student in a group must write his or her own paper. The final project must be turned in by May 19, the last day of the final exam week. The term paper should be around 15-20 typed, double-spaced, pages.

Their weights in your grade are 10% for each short assignment and assignment and 30% for the presentation, and 30% for the final paper.  

All assignments will be submitted by email, as a Word, WordPerfect, or Powerpoint document.

Assignments are due by midnight Friday of the week they scheduled to be turned in.

 

Grading:  Assignments will be graded on a 0 to 4 point scale where:

4=Outstanding (Good effort, on target, and complete performance)

3=Good (Good effort shown in work.  Only minor problems in performance)

2=Acceptable (The performance is moderately flawed, but is acceptable.)

1=Unacceptable (The performance is not indicative of graduate level work, is severely flawed or is indicative on a substandard level of effort.)

0=Assignments that are not turned in or are indicative of such bad performance that they should not have been turned in.

Before each assignment, I will tell you what I need to see to attain a 4.  Generally, it requires showing how certain concepts we have read about and discussed in class are to be applied. 

Final Grades will be calculated by weighting each individual assignment grade according to the weights defined in the previous section.  Numeric grades will be translated as follows:

A= 3.50001 - 4.0 

B= 2.75001 - 3.5

C= 1.75001 - 2.75

D= 0.75001 - 1.75

F = 0.00000 - 0.75

Please note that there will be no opportunity for “extra credit”.  Grades will be determined by the scheme outlined above - no exceptions.

Late Assignments:  Homework should be submitted by midnight Friday of the week it is due unless special arrangements are made with me in advance.  Moreover, special arrangements will only be instituted under extenuating circumstances, and thus, assignments should generally be turned in on time.  If an assignment is turned in late without advanced approval, then the grade received for the assignment will be automatically decreased by 1 point for each week or portion thereof it is late.

Grades of “Incomplete”:  A grade of incomplete will generally not be allowed except in cases of extreme hardship. 

Honor System:  Each student is expected to complete all assignments independently except as otherwise allowed--e.g., with my prior approval, working in groups on papers or projects.  There will be plenty of opportunity in class to discuss your projects and assignments, and these discussions may continue outside class.  However, the write-up must be your own work, and you are expected to show that you understand it in your in-class summary of the assignment.  If you need help with some aspect of your assignment, then come to me and I will assist you to a reasonable extent. 

Accommodations:  If you need academic accommodation by virtue of a documented disability, please contact me as soon as possible to discuss your needs.  Students with documented needs for such accommodation must meet the same achievement standards required of all other students, although the exact way in which achievement is demonstrated may be altered.  If you would like academic accommodation by virtue of your religion (e.g., turning in homework at a time other than the due date because that date falls on a religious holiday), then please contact me as soon as possible to discuss your request.  All requests for academic accommodations should be made within two weeks of the start of class.

Auditors:  For individuals who are auditing the course can attend class as much or little as they would like, reading the assigned material and performing the homework problems is encouraged, but not required. 

 

Course materials:  The readings for this course are a textbook and a set of research reports, journal articles, and chapters from edited volumes, all of which but three are available on the internet (see below).  Packets containing the three hardcopy readings are available from the EDMS office in Benjamin 1230.

Textbook

Irvine, S.H., & Kyllonen, P.C. (Eds.), Item generation for test development. Hillsdale, NJ: Erlbaum. ISBN 0-8058-0593-1

 

Papers available in hardcopy

Baxter, G. P., Elder, A. D., & Glaser, R., (1996). Knowledge-based cognition and performance assessment in the science classroom.  Educational Psychologist, 31 (2), 133-140.

Hively, W., Patterson, H.L., & Page, S.H. (1968).  A "universe-defined" system of arithmetic achievement tests.  Journal of Educational Measurement, 5, 275-290.

Katz, I. R. (1994).  Coping with the complexity of design: Avoiding conflicts and prioritizing constraints.  In A. Ram, N. Nersessian, & M. Recker (Eds.), Proceedings of the Sixteeth Annual Meeting of the Cognitive Science Society.  Hillsdale, NJ: Erlbaum.

 

Papers available on the Web

Almond, R.G., Steinberg, L.S., & Mislevy, R.J. (2002).  Enhancing the design and delivery of assessment systems: A four-process architecture.  Journal of Technology, Learning, and Assessment, 1(5) [online journal]. http://www.bc.edu/research/intasc/jtla/journal/v1n5.shtml

Mislevy, R.J. (in press).  Substance and structure in assessment arguments. Law, Probability, and Risk.  Available as CSE Technical Report 605.  Los Angeles, CA: Los Angeles:  The National Center for Research on Evaluation, Standards, Student Testing (CRESST), Center for Studies in Education, UCLA. http://www.cse.ucla.edu/reports/R605.pdf

Mislevy, R.J., Almond, R.G., & Lukas, J.F. (2003). A brief introduction to evidence-centered design. CSE Technical Report /  ETS Research Report RR-03-16.  Los Angeles, CA: Los Angeles:  The National Center for Research on Evaluation, Standards, Student Testing (CRESST), Center for Studies in Education, UCLA.  http://www.education.umd.edu/EDMS/mislevy/papers/BriefIntroECD.pdf

Mislevy, R.J., Steinberg, L.S., & Almond, R.A. (2002).  Design and analysis in task-based language assessment. Language Assessment, 19, 477-496. Available as CSE Technical Report 579.  Los Angeles, CA: Los Angeles:  The National Center for Research on Evaluation, Standards, Student Testing (CRESST), Center for Studies in Education, UCLA. http://www.cse.ucla.edu/reports/TR579.pdf

Mislevy, R.J., Steinberg, L.S., & Almond, R.A. (2003).  On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3-66. Available as CSE Technical Report 597.  Los Angeles, CA: Los Angeles:  The National Center for Research on Evaluation, Standards, Student Testing (CRESST), Center for Studies in Education, UCLA. http://www.education.umd.edu/EDMS/mislevy/papers/structure.pdf

Mislevy, R.J., Steinberg, L.S., Breyer, F.J., Almond, R.G., & Johnson, L. (1999).  A cognitive task analysis, with implications for designing a simulation-based assessment system.  Computers and Human Behavior, 15, 335-374. Available as CSE Technical Report 487.  Los Angeles, CA: Los Angeles:  The National Center for Research on Evaluation, Standards, Student Testing (CRESST), Center for Studies in Education, UCLA. http://www.cse.ucla.edu/Reports/TECH487.PDF

Mislevy, R.J., Steinberg, L.S., Breyer, F.J., Almond, R.G., & Johnson, L. (2002).  Making sense of data from complex assessments.  Applied Measurement in Education, 15, 363-378. Available as CSE Technical Report 538.  Los Angeles, CA: Los Angeles:  The National Center for Research on Evaluation, Standards, Student Testing (CRESST), Center for Studies in Education, UCLA. http://www.education.umd.edu/EDMS/mislevy/papers/MakingSense.pdf

 

Web sites

WebLAS  http://www.weblas.ucla.edu/

This is the central works-in-progress site for the Web-based Language Assessment System (WebLAS) project at UCLA. The material here represent collaboration efforts of the Principal Investigator, Dr. Lyle Bachman, the Language Subject-Matter Experts, and the WebLAS Software Development Team in a joint conjunction between Applied Linguistics and the Center for Digital Humanities.

GLOBE  http://globeassessment.sri.com/

The GLOBE Classroom Assessment site, designed and developed by the Center for Technology in Learning (CTL) at SRI International, provides sample student assessment tools and frameworks to provide teachers and students with evidence about progress on GLOBE program goals.