EDMS 738: Bayesian Inference and Measurement Models

EDMS 738: Seminar in Special Problems in Measurement

Fall 2007

Bayesian Inference and Measurement Models

Syllabus

Prof. Robert Mislevy

1230-C Benjamin Building

rmislevy@umd.edu

Course Description: This course begins with an overview of the concepts and methods of Bayesian inference, with a particular emphasis on educational and psychological measurement models. Issues of estimation for individual students and population structures will be addressed. The methods on which attention will be focused are discrete Bayesian inference networks and Markov Chain Monte Carlo (MCMC) estimation. Models we will address include classical test theory, item response theory, latent class models, factor analysis, and cognitive diagnosis.

In this class the student will learn to use the computer programs MSBNx and WinBUGS.

Prerequisites: The prerequisites are EDMS 623, EDMS 646, and one of the following: EDMS 657 (factor analysis), EDMS 723 (item response theory) or EDMS 724 (latent class models), or instructor permission.

Course Evaluation:

There are six assignments:

An ungraded ‘start-up assignment’. The ungraded start-up assignment is a paragraph about yourself. The objective is for me to get to know you so I can aim assignments, talks, and discussions appropriately. Tell me a bit about your educational and work background, including measurement and statistics, and experiences with assessment.
Short assignments or question sets. Based on readings or continuations of work done in class.
A term paper. Based on assessment, research paper, or project of the student’s own choosing, carry out a Bayesian analysis of real or simulated data. Students will briefly present their problem to the class earlier in the semester. Woring in groups of two or three on a shared example is encouraged but not required. However, each student in a group must write his or her own paper. The final project must be turned in by the last day of the final exam week. The term paper should be around 10-15 typed, double-spaced, pages.

Homework is to be turned it. Sample responses will be posted on the class web site and discussed in class. Their weights in your grade are 50% for satisfactory completion of homework assignments and 50% for the final paper.

All assignments will be submitted by email, as a Word, WordPerfect, or Powerpoint document.

Assignments are due by midnight Friday of the week they scheduled to be turned in.

Grading: Assignments will be graded on a 0 to 3 point scale where:

3=Good (Good effort shown in work. No problems or minor problems in performance)

2=Acceptable (The performance is moderately flawed, but is acceptable.)

1=Unacceptable (The performance is not indicative of graduate level work, is severely flawed or is indicative on a substandard level of effort.)

0=Assignments that are not turned in or are indicative of such bad performance that they should not have been turned in.

Final Grades will be calculated by weighting each individual assignment grade according to the weights defined above. Numeric grades will be translated as follows:

A= 2.51 - 3.0

B= 2.01 - 2.5

C= 1.51 - 2.0

D= 0.71 - 1.50

F = 0.00000 - 0.70

There will be no opportunity for “extra credit”. Grades will be determined by the scheme outlined above.

Late Assignments: Homework should be submitted by midnight Friday of the week it is due unless special arrangements are made with me in advance. Special arrangements will only be instituted under extenuating circumstances, and thus, assignments should generally be turned in on time. If an assignment is turned in late without advanced approval, then the grade received for the assignment will be automatically decreased by 1 point for each week or portion thereof it is late.

Grades of “Incomplete”: A grade of incomplete will generally not be allowed except in cases of extreme hardship.

Honor System: Each student is expected to complete all assignments independently except as otherwise allowed--e.g., with my prior approval, working in groups on papers or projects. There will be opportunity in class to discuss your projects and assignments, and these discussions may continue outside class. However, the write-up must be your own work, and you are expected to show that you understand it.

Accommodations: If you need academic accommodation by virtue of a documented disability, please contact me as soon as possible to discuss your needs. Students with documented needs for such accommodation must meet the same achievement standards required of all other students, although the exact way in which achievement is demonstrated may be altered. If you would like academic accommodation by virtue of your religion (e.g., turning in homework at a time other than the due date because that date falls on a religious holiday), then please contact me as soon as possible to discuss your request. All requests for academic accommodations should be made within two weeks of the start of class.

Auditors: For individuals who are auditing the course, they can attend class as much or little as they would like, and reading the assigned material and performing the homework problems is encouraged, but not required.

Course materials: The readings for this course are a textbook, selections from books in progress that will be made available on the web, and a number of articles/research-ports that are available without charge on the web.

Text: Selections from books-in-progress (provided online)

Almond, Mislevy, Williamson, Yan, & Steinberg (in progress). Bayes nets in education assessment. New York: Springer-Verlag.

Mislevy, Mazzeo, Lim, & Kulick (in progress). Design, analysis, and reporting in large-scale assessment. New York: Springer-Verlag.

Levy, R. (in progress).Bayesian inference in educational measurement.

Articles / Research Reports

Edwards, W. (1998). Hailfinder. Tools for and experiences with Bayesian normative modeling. American Psychologist, 53, 416 – 428. [password required]

Mislevy, R.J. (1994). Evidence and inference in educational assessment. Psychometrika, 59, 439-483. Online version available as http://www.cse.ucla.edu/products/Reports/TECH414.pdf

Mislevy, R.J., & Gitomer, D.H. (1996). The role of probability-based inference in an intelligent tutoring system. User-Modeling and User-Adapted Interaction, 5, 253-282. Online version available as http://www.cse.ucla.edu/products/Reports/TECH413.pdf

Mislevy, R.J., Wilson , M.R., Ercikan, K., & Chudowsky, N. (2003). Psychometric principles in student assessment. In T. Kellaghan & D. Stufflebeam (Eds.), International Handbook of Educational Evaluation (pp. 489-531). Dordrecht , the Netherlands : Kluwer Academic Press. Online version available as http://www.cse.ucla.edu/products/Reports/TR583.pdf

Mislevy, R.J., Steinberg, L.S., Breyer, F.J., Almond, R.G., & Johnson, L. (2002). Making sense of data from complex assessments. Applied Measurement in Education, 15, 363-378. Online version available as http://www.cse.ucla.edu/products/Reports/TECH538.pdf

Sinharay, S. (2003). Assessing convergence of the Markov chain Monte Carlo algorithms: A review. Research Report RR-03-07. Princeton , NJ : Educational Testing Service. Online version available as http://www.ets.org/Media/Research/pdf/RR-03-07-Sinharay.pdf

Sinharay, S., & Johnson, M. (2003). Simulation studies applying posterior predictive model checking for assessing fit of the common item response theory models. http://www.ets.org/Media/Research/pdf/RR-03-28-Sinharay.pdf

Calendar

EDMS 738: Bayesian Inference & Measurement Models	Intended Schedule, as of September 31, 2007
Fall 2007	Robert J. Mislevy

Class/Date

Topic

Readings

Assignment

Due Friday

9/10

Introductions & Overview;

ECD models

"Psychometric Principles" pp. 1-25 BDA, Sections 1.1-1.6,1.8

BDA, Sections 1.7,1.9-1.11

BEIM, Ch 1

1 paragraph description of yourself

9/17

Probability concepts; Intro to Bayes nets

Download MSBNx

DAR, Section 2.5

“Evidence & inference” pp. 1-45

Probability review: DAR, 2.1- 2.5; BNEA, Ch. 2; BEIM, Ch 2

9/24

Propagation in Bayes nets

Bayes nets examples

BNEA, 5.1& 5.2 ; 5.3 & 5.4

Edwards (1998) esp. 420-426

Re graphical models: BNEA, Ch. 4

Bayes net problems

10/1

Cognitive diagnosis

“Role of prob-based inference in an ITS”

BNEA, Ch. 6; Ch 8

“Making sense of data…”

Download WinBUGS

10/8

General Bayesian model / MCMC estimation 1

BNEA, 9.1; BUGS tutorial

BDA 2.6, 3.1-3.4, 14.1-14.2

BEIM Ch 3-5

10/15

MCMC estimation 2

Student problem presentations begin

BNEA, 9.2, 9.3, & 9.5

Sindharay

BDA 11.1-11.6

BUGS problems

10/22

Classical Test Theory 1

"Psychometric Principles" pp. 26-38

DAR, Ch. 3, pp. 1-10; BEIM Ch 7

10/29

Classical Test Theory 2 Preposterior distributions

DAR, Ch. 3, pp. 10-21

Sindharay & Johnson; BDA 6.1-6.5, 6.7

DAR, Ch. 3, pp. 21-32

CTT assignment

11/5

IRT 1

"Psychometric Principles" pp. 41-45

DAR, Ch. 4, pp. 1-10

#10

11/12

IRT 2

"Psychometric Principles" pp. 45-47 BEIM Ch 9

IRT assignment

#11

11/19

Latent class analysis

BNEA, Ch 9, Sec 9.2, 9.3, & 9.5

BEIM 10

#12

11/26

Factor analysis

BEIM 8

#13

12/3

Generalizability Theory

"Psychometric Principles" pp. 39-40

#14

12/10

Missing data

BDA, Ch 21

Term Paper Due

12/19

Notes

For readings, bold indicates primary readings; others are supplemental.

Page numbers for articles refer to online research report versions.

BDA = Gelman, Carlin, Stern, & Rubin’s Bayesian Data Analysis, 2nd edition.

BNEA = Almond, Mislevy, Steinberg, Williamson, & Yan’s Bayes nets in educational assessment.

DAR = Mislevy, Mazzeo, Lim, & Kulick’s Design, analysis, and reporting in large-scale assessment.

BIEM = Levy's Bayesian inference in educational measurement.