December 6, 2007

Plans for class on Dec 9: Mixture IRT models.

• Overheads for class presentation
• Paper that goes along with the overheads (optional)
• Paper on the Mixture Rasch Model by Ru Lu, including BUGS setup
• Excel spreadsheet for generating dichotomous data uner the Rasch model
• WinBUGS file for Rasch analysis of one 150x12 matrix of responses
• WinBUGS file for mixture of two Rasch models for 300x12 matrix responses (actually the catenation of two 150x12 matrices generated with different Rasch models, but BUGS doesn't know this)

December 1, 2007

Up on the class website: Information about the course project.

November 30, 2007

Plans for class on Dec 3:

• Finish latent class models -- looking at Dr. Dayton's "cheating" data with unconstrained and constrained BUGS models.
• Start one dimensional factor analysis (here's the BUGS setup)

Re projects: Dr. Larwence Rudner offers a free 3PL IRT estimation program he calls PARAM-3PL. You may want to try it. (He also has a nice tutorial on adaptive testing with IRT.) Here is the sample data set he provides with it. It is a large data set; for running BUGS, you may want to use a subset (e.g., 500 examinees, 20 items). You will need to reformat the data to run in BUGS. Note that there are some empty spaces, which need to become NA for BUGS. Here are some kinds of project ideas you can use with this data set (or adapt the ideas to other data sets):

• Dr. Rudner provides thetas in this data set. Run IRT estimation in the usual way, with thetas unknown, and again with the thetas read in as known data. Compare the results for item parameters (especially posterior standard deviations).
• Compare results with the same items but different numbers of examinees -- say 100, 250, 500, and 1000.
• See if you can get PARAM-3PL to run, and compare its results for a BUGS run on the same data.
• Run BUGs four times with the same model and items, but with different examinee data -- say examines 1-500, 501-1000, 1001-1500, and 1501-2000. Compare and discuss your results.
• Fit the Rasch, 2PL, and 3PL models to the same subset of data, and compare the results.

November 26, 2007

Another try on example 2 for graded response IRT model -- 500 subjects, 10 items.

November 23, 2007

Plans for class on Nov 26: Presentation on graded response IRT models, with accompanying calculation of probabilities and a BUGS examples (example 1 and example 2). If time, start to look at latent class models.

November 21, 2007

An excellent resource on the Bayesian concept of exchangeability and its role in Bayesian inference (password protected, for registered UM students only).

November 20, 2007

The Spiegelhalter et al. (2002) paper on the DIC (password protected, for registered UM students only).

For course evaluations this semester, we will have both the familiar paper ones to fill out in class and the new on-line ones you do on your own. For the paper ones, I assume this will happen like it has in the past: They give the packet to me, I bring it to class, and have a volunteer manage the process while I head home a little early. For the on-line ones, you will receive an e-mail notice prior to the start of the evaluation process.  The anticipated dates for the on-line course evaluation system to be open are Tuesday, November 27 through Wednesday, December 12.   Students will receive an e-mail notice via their official university e-mail address at the start of the on-line evaluation process.  For more detailed information regarding the on-line course evaluation process, see
https://www.irpa.umd.edu/Assessment/crs_eval.shtml.

November 18, 2007

Assignment 5, on IRT, is now up. It is due Nov 30.

Plans for Monday's class:

• The DIC index for comparing model fit. CTT-DIC1, CTT-DIC2, CTT-DIC3, CTT-DIC4 are WinBUGS setups for classical test theory model comparisons (.odc files -- need to save to your drive, then open from within WinBUGS).
• WinBUGS for a three-parameter logistic (3PL) model.
• Talk about Assignment 4?

November 16, 2007

I am working on Assignment 4's, and plans for Monday's class. There are now comments on the solutions.

November 12, 2007

Thanks for Assignment 4's. I will get to them this week.

For in-class work today, PPMC with CTT.

November 5, 2007

In the November 12 class meeting the lecture will be Item Response Theory. WinBUGS files are up for a Rasch model example and a three-parameter logistic (3PL) model. If time allows, we will talk about the DIC index for comparing model fit.

October 31, 2007

Optional Homework Assignment 3 1/2 is now up -- the Normal-Normal model.

Plans for Nov 5: Finish up classical test theory. Class calendar has been revised to reflect this additional week on CTT.

October 23, 2007, update

A good article on dealing with label switching in MCMC is Chung, Loken, Schafer's (2004) paper in American Statistician, "Difficulties in drawing inferences with finite-mixture models: A simple example with a simple solution." Here is a password-protected copy for registered students.

October 23, 2007

Here is my WinBUGS input on multiple regression, with the intercept missing. I put up a .txt rather than an .odc format, so you don't have to open it from WinBUGS.

Homework Assignment 3 due Friday.

Next Monday: CTT continued: Overheads CTT Part 2 and CTT Part 3; WinBUGS CCT example.

October 18, 2007

The next 2-3 meetings will focus on classical test theory. The overheads are CTT Part 1, CTT Part 2, and CTT Part 3. Here are associated WinBUGS files: CTT example, CTT-PPMC. The associated readings are pp 26-38 of Psychometric Principles and, for fuller details of what we'll be talking about in the overheads, Chapter 3 of Design, analysis, and reporting in large-scale assessment, with its accompanying figures and tables (recall that these files require a password to open).

After the lecture, we will do an in-class WinBUGS example on multiple regression. The data set we will analyze is from Prof. David Lane's nice online introductory statistics website Hyperstat.

Assignment 4 on classical test theory is now up on the website. It is due November 9. The calendar has been revised to push both this assignment and the IRT assignment a week into the future.

October 13, 2007

An excellent short article entitled "Why environmental scientists are becoming Bayesians," byJames S. Clark. Turns out it is pretty much for the same reason measurement scientists are. The first half of the article lays out the hierarchical nature of problems and the interest in modeling understood effects and variation associated with not-understood effects -- exactly the kind of issues we are moving into in our class.

October 12, 2007

Assignment 3 is now up. It is due Oct 26.

Plans for class meeting on Monday, October 16:

• Presentation MCMC estimation and a general model for Bayesian inference with measurement models.
• Working through WinBUGS for binomial data.
• If time, MSBNx and WinBUGS models for estimating proficiency in taking patient histories. Files for WinBUGS models for estimating proficiency in taking histories:
  • Proficiency1: 100 cases, Proficiency observed directly
  • Proficiency2: 100 cases, Proficiency observed directly + three History-taking observations
  • Proficiency3: 100 cases, Proficiency not observed + three History-taking observations

  The Netica file I used to generate the data is here.

Additional resources:

• Chapters on Bayesian inference and measurement models by Prof. Roy Levy of Arizona State University (and proud--I think--EDMS alumnus). The files are password protected, but students registered in the class know what the password is. These are chapters in a forthcoming book that parallel and expand on the overheads used in the class lectures. At this point in our class, see especially the ones on MCMC estimation and WinBUGS.

October 7, 2007

Bayes net for Item 1, parts g-i, of Assignment 2. Remember to save the file to your computer, and open it from MSBNx. This extended netowrk has five copies of Cathy. Watch what happens to the nodes for Coin, Ann, and Bob when you enter four Heads and one Tails for the Cathy nodes (and if you have more energy than me, you could try 100 Cathy nodes and enter Heads for 80 and Tails for 20 of them).

October 6, 2007

Sample answers for Assignment 2.

October 4, 2007

Plans for class meeting on Monday, October 8: Presentation more on Bayesian inference. First look at WinBUGS; bring your notebook; download WinBUGS before class if you can.

September 27, 2007

A couple of new Bayes nets examples:

• A MSBNx network for Tatsuoka's mixed number subtraction, an example of cognitive diagnosis we will discuss on October 1. It is discussed in the Evidence and Inference in Educational Assessment reading.
• FYI, here is a Netica instantiation of the Hailfinder example from Edwards, W. (1998). I believe you can open this with the free version of Netica.

Also FYI, the first author of of text, Andy Gelman, has an interesting blog on statistics in the social sciences.

September 25, 2007

Due date for Assignment 2 is October 5.

Plans for October 1:

• Working together on a set of Bayes nets problems in small groups.
• Presentation on Cognitive diagnosis.

September 24, 2007

Throat clinic Bayes net problem

September 13, 2007

Further thoughts on classes September 17 and Sept 24: We will probably not finish both sets of overheads and the intro to MSBNx on Sept. 17, so we will continue these topics on Sept. 24. Further, on Sept 24, we will spend time in class working together on a set of Bayes nets problems in small groups.

September 13, 2007

For class on Monday, September 17: We will talk about probability concepts with discrete variables and Bayes nets (the hot links are to the overheads for these topics). The reading “Evidence & inference” pp. 1-45 provides the background you need, but see the additional readings on the calendar for more if you'd like more.

• Friday: Email me Assignment 1, the paragraph about you.
• Download MSBNx before class Monday, and bring it with you if you have a notebook computer. We will use it for some examples in class, again in small groups.
• Here are some MSBNx examples. Either download these as text files and open them through MSBNx, or set your computer so the default program to open .xbn files is MSBNx. Otherwise, if you just open the file you'll see the html rendition of it--which is interesting in itself and we may talk about briefly.

August 18, 2007

Welcome to EDMS 738 for Fall 2007, "Bayesian inference and measurement models"! Our class will meet Mondays 4:15-7:00 Eastern time, in the Engineering Annex Building 0307.

The first meeting is September 10. The first half of the class will be introductions and an overview of the course. After a break will be an initial presentation on the basic models of the evidence-centered design framework, which we will be using throughout the course to organize our thinking. If you are reading this note before our first class meeting, read pp. 1-25 of this research report from our reading list before class: Psychometric principles in student assessment.

On the class website you will find a link to overheads for the class, which now contains overheads for the first class meeting: Key ideas and Intro to Evidence-Centered Assessment Design. Once we are up and running I will aim to get overheads onto the website by end of day Friday before a class meeting. This will allow you to download and print copies of overheads if you wish, so you have copies you can make notes on during presentations and discussions. Overheads that are powerpoint slides such as these, you may want to download to your computer and print them out 2, 3, or 6 per page to save paper.

Feel free to contact me if you have any questions. My email address is rmislevy@umd.edu.

Bob Mislevy