December 1, 2004

Assignment 7 is now up. It is due a week from Friday, namely December10--the last day of the semester before finals week.

November 28, 2004

Hope everybody had a good Thanksgiving holiday! We are back now, and into the home stretch of the semester.

For Monday, November 29:

Finally we look at the WinBUGS Example based on the Two Skills problem of Assignment 3 : Doodle and Code.

We will look at a classical test theory example using posterior predictive distributions to examine test fit.

Maybe, depending on time, look at a sequence of models leading to one-dimensional factor analysis (overheads). Bugs setups for this presentation are simple regression, standardized regression, multivariate standardized regression, and one-D factor analysis.

November 17, 2004

Overheads re latent class models for November 22 class meeting are now available. So are the WinBUGS files for examples we will look at in connection with the presentation:Univariate manifest model, Multivariate manifest model, Latent class model with Unif(0,.5) prior, and Latent class model with Unif(0,1) prior.

We also will see if we can look at the WinBUGS Example based on the Two Skills problem of Assignment 3 : Doodle and Code.

November 14, 2004

WinBUGS Example based on the Two Skills problem of Assignment 3 now available: Doodle and Code (.odc files--save as file and open with WinBUGS). If there is time, we might look at this Nov 15.

November 10, 2004

Re the IRT homework: BUGS wants you to run at least 4001 cycles before the 'set' button on the DIC dialogue box is activated.

November 9, 2004

Here is Yunyun's slide with references for Wainer's IRT model for conditional dependence in testlets. You can download pdf copies of research reports on the use of this approach from the ETS Research Reports site: RR-02-02 and RR-01-09. He also has a computer program called ScorRight for estimating the parameters in the model, using MCMC; the User Guide is available there as RR-01-06.

November 3, 2004

In the November 8 class meeting the lecture will be Item Response Theory. WinBUGS .odc files are up for a Rasch model example and a three-parameter logistic (3PL) model.

Up on the website is Assignment 6, which concerns IRT. It is due Friday Nov 19.

Looking ahead, we will have a guest presenter Nov 15: Roy Levy, to speak to us about model fit in MCMC, in the context of IRT. Read the following paper from our reading list: Sinharay, S., & Johnson, M. (2003).  Simulation studies applying posterior predictive model checking for assessing fit of the common item response theory models. http://www.ets.org/research/dload/RR-03-28.pdf

November 1, 2004

In the November 1 class meeting the lecture will be Classical test theory, Part 3. I have received your emails with Assignment 5 results. I will work through them and get back to you individually over the next couple days. We will talk about these problems in class, either today (I hope) or next week, because they introduce the idea of posterior predictive distributions and a key tool for model criticism in Bayesian MCMC estimation.

October 19, 2004

Now up on the class site are a BUGS file for the upcoming in-class example (save it to your computer rather than just clicking on the link) and Assignment 5, which is based on this example. Note that the example uses matrix format for reading in data, so you will need to highlight and click data twice, for the list portion and the matrix portion of the data. (What would happen if you only read in the list portion?)

October 15, 2004

The next 2-3 meetings will focus on classical test theory. The overheads are CTT Part 1, CTT Part 2, and CTT Part 3. The associated reading is Chapter 3 of Design, analysis, and reporting in large-scale assessment, with its accompanying figures and tables (recall that these files require a password to open).

After the lecture, we will look at a WinBUGS example for Bayesian inference about the mean of a normal distribution. Here is a doodle and examples of BUGS code for this problem (remember not to just click on these files; save them to your computer and open them from within WinBUGS).

During next week I will put up another assignment based on classical test theory, which will be due October 29.

October 13, 2004

Hint for Assignment 4: Specifying parameters for a categorical distribution.

October 11, 2004

Corrected link to a generic measurement model & MCMC estimation overheads used in the October 11 presentation.

October 10, 2004

Assignment 4, generating distributions with WinBUGS, is now up. It is due Monday October18. Note that this due date is not the usual Friday, since I did not post this problem until Monday the 11th.

Re the question that came up in last week's class about a large number of ratings of independent History-performances by the same biased judge: The answer is that although every rating will be biased, if you know the conditional probabilities that tell you just how this judge is biased (and assume the ratings are independent, all with these same conditional probabilities), then as the number of performances & ratings increase you can infer the proportion of performances that were truly adequate and inadequate. In this example we also knew the conditional probabilities of adequate and inadequate performances from experts and novices, so with increasing numbers of independent performances and independent but identically biased ratings we would approach certainty about the examinee's expertise. Here is an MSBNx setup with ten performances and ratings.

October 8, 2004

Plans for October 11 meeting:

• Presentation on a generic measurement model & MCMC estimation
• Some examples using BUGS. New link on the class website, where we will make sample WinBUGS input files.

September 29, 2004

Plans for October 4 meeting: Presentation on more on Bayesian inference; Marc will tell us about the context of his project (overheads and/or links when they become available); looking ahead toward BUGS.

September 24, 2004

The next class meeting is Monday, September 27. We will discuss Bayes nets further, including a discrete IRT example and a presentation and an in-class example on cognitive diagnosis. Toshi will present the background and initial ideas for his class project.

Assignment 3, "More Bayes nets," is now up. It is due Friday October 1.

September 22, 2004

Typos corrected in Assignment 2; link to Assignments added to EDMS 738 home page; link to BobCoin MSBNx file points to the right place.

September 15, 2004

The next class meeting is Monday, September 20, 2004. We will discuss Bayes nets (overheads). One of the examples we will look at in more detail is the widely-used "Asia" example. An MSBNx input file for this example is here. A short & readable discussion of this example can be found here at the website for the Nettica Bayes net engine. (Nettica has some nice features that MSBNx doesn't have, but it costs money to get. However, you can run small problems with the free version, and from time to time we may use it to demonstrate some ideas.)

Assignment 2, Bayes nets problems, is now up on the class web site at the new Assignments page. It is due Friday, October 1, so if there are questions we can talk about them in class September 20 & 27.

The password for the Coin Flip overheads is NOTEDMS738. It is still a read-only file, so when you try to open it, type in this password on the first password diaglogue box that pops up; another one pops up, and you click the 'read only' button. An MSBNx file for my unreliable report is here.

September 3, 2004

The first class meeting was Monday, August 30, 2004. We discussed the key ideas we will address in the course, and took a brief look at evidence-centered assessment design and the role measurement models play in an assessment argument.

There will not be an ETS meeting site this semester.

Plans for the next class meeting, on September 13:

1. Coin toss discussion. (These overheads are password protected until class, so as not to give away the ending.)

2. Lecture, Introduction to probability.

3. Building some simple Bayes nets with MSBNx.

4. Depending on time, maybe start on Bayes nets lecture.

A link with MSBNx examples is now up on the class website.

August 28, 2004

Now up on the class website: A link to overheads for the class, and overheads for the first class meeting: Key ideas and Intro to evidence-Centered Assessment Design. Once we are up and running I will try to get overheads onto the website by end of day Friday before a class meeting. This will allow you to download and print copies of overheads if you wish, so you have copies you can make notes on during presentations and discussions. Overheads that are powerpoint slides such as these, you may want to download to your computer and print them out 2, 3, or 6 per page to save paper.

August 17, 2004

Welcome to EDMS 738 for Fall 2004, "Bayesian inference and measurement models"! Our class will meet Mondays 4:15-7:00 Eastern time, provisionally at three locations: at the University of Maryland, in Engineering 3103, at the Educational Testing Service, Princeton NJ, the room to be determined, and SRI International, in Menlo Park, CA. We will be hooked up by live teleconference, so we can have interactive discussions and presentations across sites.

The first meeting is August 30. The first half of the class will be introductions and an overview of the course. After a break will be an initial presentation on the basic models of the evidence-centered design framework, which we will be using throughout the course to organize our work.If you are reading this note before our first class meeting, read this research report from our reading list before class: A brief introduction to evidence-centered design, by Mislevy, Almond, & Lukas.

Feel free to contact me if you have any questions. My email address is rmislevy@umd.edu.

Bob Mislevy