André A. Rupp
Dr. André A. Rupp is an Associate Professor in the Measurement, Statistics, and Evaluation (EDMS) program in the Human Development and Quantitative Methodology (HDQM) department at the University of Maryland. His synthesis-oriented work frequently circumscribes, deconstructs, and re-arranges the current state-of-the-art of methodological research and practice at the intersection of educational and psychological measurement, applied cognitive psychology, and the learning sciences. His research interests center around cognitively-grounded assessment approaches and associated statistical models, which broadly fall under the umbrella terms diagnostic measurement / cognitively diagnostic assessment and diagnostic classification models (DCMs) / cognitive diagnosis models. To this end, he works on research around model criticism and refinement through simulation studies and applied work. From an interdisciplinary perspective he is currently most interested in exploring how an evidence-centered design framework for assessment design, scoring, and reporting as well as associated use models can be put to practical use for researchers who are interested in creating integrated diagnostic assessment systems. He is particularly engaged in helping to create diagnostic feedback based on process and product data from digital learning environments, which requires the integration of tools from multivariate statistics, modern psychometrics, and educational data mining. He currently collaborates with researchers at CISCO, the University of Wisconsin-Madison, the Friday Institute for Educational Innovation at North Carolina State University, and the Education Development Center on such projects.
Special Issue on Diagnostic Measurement in Digital Learning Environments in the Journal of Educational Data Mining (JEDM)
I am the guest co-editor of a special issue of the Journal of Educational Data Mining (JEDM) with the topic 'Diagnostic assessment in digital learning environments: An overview of the current state-of-the-art'. The aim of the special issue is to showcase the challenges and solution pathways for providing formative diagnostic feedback via embedded assessments within innovative digital learning environments. The emphasis is not on presenting seemingly "perfect" solutions for such contexts but, rather, to showcase different worked exemplars and stimulate an honest, methodologically rigorous, and forward-looking dialogue about work in this area. To provide a coherent organization of information in the special issue, a coherent principled design framework known as evidence-centered design (ECD) will be used. If you are interested in submitting your work and are just now becoming aware of this opportunity, please do not hesitate to contact me as there might always be a second follow-up issue!
Diagnostic Measurement Workshops at the University of Maryland
Every year, I hold at least one two-day workshop based on my recently published co-authored book Diagnostic measurement: Theory, methods and applications (see information below).The last two-day workshop was held August 2 and 3, 2012, at the University of Maryland in College Park, MD. Please click here for the flyer for this workshop, which includes registration information and logistical information. Look out for information on the summer 2013 workshop in the spring!
Related to this workshop you may also find the following software pacakages handout from an AERA 2009 session useful where I asked different software and code developers to describe their particular programs for specifying and estimating diagnostic classification / cognitive diagnosis models. It is now a bit dated but several of the key features of many programs are likely still the same. You may also find the program-specific references and contact information useful!
All of the workshop slides, supplementary handouts, and files for a supplementary example are available below. Additional routines for creating Mplus input files, parsing Mplus output files, and computing RMSEA & MAD fit statistics in R will be made available on this website when they are ready for distribution.
2012 Annual Meeting of the National Council on Measurement in Education (NCME) in Vancouver, BC, Canada
I was the co-chair of the NCME 2012 Annual Meeting with my colleague Dr. Joanna Gorin at Arizona State University - we had a wonderful time in beautiful Vancouver, British Columbia, Canada! If you are interested in taking a look at the Annual Meeting program just click here.
If you attended session L1 entitled 'Embedded assessment within innovative digital learning environments: A methodological case study' on Monday morning from 10:35 - 12:05, you can download the paper that describes most of the work in this session here.
Selected Recent Publications
1. Rupp, A. A. (2013). Clustering and classification. In Little, T. (Ed.), The Oxford handbook of quantitative methods (pp. 517-550). New York: Oxford University Press.
2. Rupp, A. A., diCerbo, K. E., Levy, R., Benson, M., Sweet, S., Crawford, A., Fay, D., Kunze, K. L., Calico, T., & Behrens, J. (2012). Putting ECD into practice: The interplay of theory and data in evidence models within a digital learning environment. Journal of Educational Data Mining, 4, 49-110.
3. Kunina-Habenicht, O., Rupp, A. A., & Wilhelm, O. (2012). The impact of model misspecification on parameter estimation and item-fit assessment in log-linear diagnostic classification models. Journal of Educational Measurement, 49, 59-81.
4. Mislevy, J., Rupp, A. A., & Harring, J. R. (2012). Detecting local item dependence in polytomous adaptive data. Journal of Educational Measurement, 49, 127-147.
5. Rupp, A. A., & Sweet, S. (2011). Analysis of multivariate social science data [Book review]. Structural Equation Modeling, 18, 1-8.
My synthesis-oriented work frequently circumscribes, deconstructs, and re-arranges the current state-of-the-art of methodological research and practice at the intersection of educational and psychological measurement, applied cognitive psychology, and the learning sciences. My current research interests center around cognitively-grounded assessment approaches and associated statistical models, which broadly fall under the umbrella terms diagnostic measurement / cognitively diagnostic assessment and diagnostic classification models (DCMs) / cognitive diagnosis models.
I have recently co-authored a book on this topic with two colleagues in the field, Dr. Jonathan Templin and Dr. Robert Henson, which has received rather positive reviews from colleagues and practitioners so far. The book's citation is Rupp, A.A., Templin, J., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications. New York: Guilford Press and has a companion website that you can access here.
I have also co-developed a workshop on this topic with my co-authors, which I held in past years at the University of Maryland; see the most recent dates as noted at the top of this page. I am generally interested in presenting this workshop upon request, both nationally and internationally so please contact me if you are interested in this workshop!
Moreover, I am currently involved in two related research strands in the area of diagnostic measurement. In the first strand, my work centers around developing embedded diagnostic assessment approaches for modeling learning progressions of complex skill sets in games- and simulation-based learning environments.The second research strand concerns describing and investigating the effects of different types of model misspecification on processes for model criticism. A few more words on both of these below!
Embedded assessment in digital learning environments
I am currently collaborating with a research team under the direction of Dr. Roy Levy at the University of Arizona as well as Dennis Frezzo, Martin Benson, and other colleagues from Cisco - as well as historically John Behrens and Kristen DiCerbo, who are now at Pearson. The project is concerned with evidence identification and accumulation for their interactive learning and assessment system called PacketTracer. In this environment, learners from literally all over the world can practice network engineering skills. The interface allows them to interactively select devices and requisite tools and allows them to manage, configure, and troubleshoot their networks in authentic vignettes. The process data in this project consists of logfiles while the product data consists of scored final network configurations. The work at the University of Maryland explores the utility of DCMs for providing multivariate score profiles based on the network configuration scores as well as how these scores can be aligned with scores from standardized assessments that the learners are taking. Moreover, we are exploring how to best tag or recode logfile entries so that they can be used as meaningful input for clustering routines and other forms of data visualization that can complement the score reports.
I am currently finishing a collaboration on the NSF-funded research grant AutoMentor: Virtual Mentoring and Assessment in Computer Games for STEM Learning, which are spearheaded by the interdisciplinary Epistemic Games Group at the University of Wisconsin at Madison (www.epistemicgames.org) under the direction of Dr. David W. Shaffer. In this project we are working on investigating methods for creating diagnostic score reports for learners in small-sample settings. The learners play various forms of so-called epistemic games in which they learn the key principles, methods, and tools of professionals in different disciplines such as journalism and urban planning. A key experience in these environments is the process of mentoring, which provides scaffolded learning experiences. Evidence for development is currently focused on various forms of discourse for which automated speech act classification, proficiency variable identification, and mentoring moves are developed at the University of Memphis under the direction of Dr. Art Graesser. The work at the University of Maryland currently centers on designing simulation studies for investigating the robustness of various methods for evidence aggregation and synthesis within these environments.
Model misspecification and criticism for DCMs
With several graduate students here at EDMS and a former graduate student in Germany I am currently investigating robustness properties of DCMs with respect to model misspecification. Work in this area overlaps with work on other latent-variable modeling frameworks since DCMs share properties with restricted latent class models, multidimensional item response theory models, and explanatory item response theory models, to name a few. Thus, I have recently worked with students on investigating the performance of item-fit statistics for bi-factor models and DCMs and have completed a synthesis paper on the design of simulation studies for person fit in item response theory. It is our intention that this line of research, in conjunction with the work of other colleagues in this area, will contribute to the creation of comprehensible reliable guidelines for DCMs that are meaningful to practitioners from diverse disciplines.
In what feels almost like another lifetime at this point, I initially studied to become a secondary school teacher for English, French, and Mathematics at the University of Hamburg in my hometown of Hamburg, Germany. Through completing my Master's work in Teaching English as a Second Language / Applied Linguistics and Mathematics / Statistics at Northern Arizona University in Flagstaff, AZ, as well as my Ph.D. work in Measurement, Evaluation, and Research Methodology at the University of British Columbia in Vancouver, BC, Canada, I got lured more deeply into the beauty of academic research and graduate-level teaching. I started my academic career at the Faculty of Education at the bilingual (English / French) University of Ottawa in Canada followed by a two-year visiting professorship at the Institute for Educational Progress in Berlin, Germany, where I worked in an interdisciplinary team on developing national standards-based assessments for English as a first foreign language.
GRADUATE STUDENTS & COLLEAGUES
Great Graduate Students
I continue to have the great fortune of being the dissertation advisor to several outstanding graduate students:
1. Olga Kunina-Habenicht, Humboldt University in Berlin, graduated summa cum laude in June 2010. She is now an Assistant Professor at the Goethe University in Frankfurt am Main in Germany. Her dissertation focused on a complex simulation study on item and model misspecification for DCMs as well as the application of DCMs to a newly developed diagnostic assessment for arithmetic in elementary school whose development she spearheaded. You can find out more about Olga's current research here and contact Olga via email here
2. Jessica Mislevy, University of Maryland (co-adviser Dr. Jeffrey Harring), graduated in May 2011. Her dissertation focused on the behavior of the Q3 and X2 model fit statistics for computer-adaptive tests, which she investigated with a complex simulation study. She also applied the statistics to the PROMIS data bank for health outcomes assessment. You can contact Jessica via email here
3. Matthew Gushta, University of Maryland, graduated in May 2012. His dissertation focused on the behavior of three model- and three-item fit statistics from confirmatory factor analysis and item response theory frameworks under correct model specification and incorrect model specification (via incorrect Q-matrix specifications). You can contact Matthew via email here.
4. Shauna J. Sweet, University of Maryland. She is a second-year Ph.D. student in our program with a previous Master of Science in Survey Methodology. She is currently working with me on the two research projects around digital learning environments outlined above. You can contact Shauna via email here.
5. Tiago Caliço, University of Maryland. He is a second-year Ph.D. student in our program and a current Fulbright scholar. He is also working with me and Shauna on the two research projects around digital learning environments outlined above. You can find Tiago's personal profile here and contact Tiago via email here
...and, of course, there are several wonderful students who have worked with me on independent studies or whom I have supervised as Master's students or certificate candidates in our department!
I have been very fortunate to work with a variety of wonderful mentors and colleagues from different disciplines in my life and feel like I never stop learning when I am in their company. So my special thanks for inspiration, engagement, and laughter explicitly goes out to the following people (in alphabetical order by last name) but it also goes out implicitly to many unmentioned others who I have met over my career so far:
- John Behrens, Cisco
- Jere Confrey, Friday Institute for Educational Innovation at North Carolina State University
- Jimmy de la Torre, Rutgers the State University of New Jersey
- Kristen di Cerbo, Cisco
- Leo Elshof, Arcadia University
- Kadriye Ercikan, University of British Columbia
- Andreas Frey, Leibniz Institute for Science and Mathematics Education
- Nicholas Gazzola, University of Ottawa
- Jim Gee, Arizona State University
- Joanna Gorin, Arizona State University
- William Grabe, Northern Arizona University
- Art Graesser, University of Memphis
- Claudia Harsch, University of Warwick
- Johannes Hartig, Universität Erfurt
- Robert Henson, University of North Carolina at Greensboro
- Kristen Huff, New York State Department of Education
- Joan Jamieson, Northern Arizona University
- Michel Knigge, Research Data Center at the Institute for Educational Progress in Berlin
- Danny Laveault, University of Ottawa
- Jacqueline Leighton, University of Alberta
- Nonie Lesaux, Harvard Graduate School of Education
- Roy Levy, Arizona State University
- Nathalie Loye, Université de Montréal
- Alan Maloney, Friday Institute for Educational Innovation at North Carolina State University
- Bob Mislevy, Educational Testing Service
- Hans Anand Pant, Institute for Educational Progress in Berlin
- David Williamson Shaffer, University of Madison at Wisconsin
- Valerie Shute, Florida State University
- Marielle Simon, University of Ottawa
- Jonathan Templin, University of Georgia at Athens
- Miriam Vock, Universität Potsdam
- Oliver Wilhelm, Humboldt University
- Bruno Zumbo, University of British Columbia
- wonderful graduate students from the past and present, and, of course,
- wonderful colleagues in my current department!
As you can imagine, I am quite happy to live not only in D.C. but so close to Broadway, "The Great White Way", in NYC, which deserves a special shoutout for all its fabulousness right here! I never knew that there were so many plays and musicals on Broadway, off-Broadway, and off-off Broadway all the time with fantastic casts until I moved to DC! So if you are interested in learning about what is happening on the Great White Way, here are a few useful sites:
1. Playbill and Broadway World for all up-to-date information about what is going on.
2. Telecharge for buying tickets to most shows - a few such as Wicked and The Lion King go through Ticketmaster (No, I don't get a commission!)
3. Theatermania, Playbill Club, and Broadway Box for discounted tickets. Note that almost all shows are on sale at some point during their runs via these legitimate outlets - no secondary market purchase necessary! A few exceptions are very popular shows such as, currently, The Book of Mormon, Wicked, or The Lion King.
You never know who you might see at a show, who might sign your Playbill or windowcard, or who poses for a picture with you...
For example, did you know that you can see Samuel L. Jackson and Angella Bassett in The Mountaintop this fall? Or Alan Rickman in Seminar? Or Jim Parsons and Ellen Barkin in The Normal Heart this summer? Or, on a different note, Andrea Bocellin in Central Park for free on September 15? Anyway, you get the idea. So book your hotel, go out, and support the performing arts! You are not too busy for that! And while you are in the Theater District, go to Trattoria Trecolori for some cozy Italian Food. Or go to Eataly if you want to venture south a bit where you can have a fantastic cappucino and amazing food!
I also generally enjoy productions by the Cirque du Soleil, in particular amazing shows like O, Totem, or Ka and have started watching a few pieces of opera such as Die Walkuere at the Met in NYC this spring. One of my favorite creator and director is Robert LePage who, incidentally, comes from Quebec, Canada. He has done many wonderful artistic things which include the creation of Totem and Ka for the Cirque du Soleil, and the current Ring cycle at the Met as well as phenomenal cinematic plays like The Blue Dragon, Lipsynch, and The Andersen Project.
Another performing arts favorite of mine is concerts and stand-up comedy shows outside of Broadway, especially when these shows are professionally produced. I have seen a lot of great acts as diverse as Michael Buble, Andrea Bocelli, Charles Aznavour, Sarah Brightman, The Rolling Stones, Paul McCartney, Matchbox 20, The Eagles, Bryan Adams, The Pet Shop Boys, Lady Gaga, K.D. Lang, Amy Grant, George Strait, Sade, The Dixie Chicks, Robin Williams, Chris Rock, Margaret Cho, John Oliver, Louis CK, and many others.
It might also interest you to learn that I have been an avid ballroom, swing, and Latin dancer in my life and have been particularly fond of West Coast Swing dancing in the DC area.
I grew up near Hamburg, Germany, which is where my mother and my father continue to live. After me being abroad for 11.5 out of 14 years now, they have had to come to terms with the fact that I am happiest where I am now. As I get older, I am trying more and more to keep a very healthy balance between investing in my career and investing in my personal life. In Washington, DC, I am now in a wonderful relationship with a woman who, trained as a clinical psychologist, is currently working as a psychology associate in the area. She is just such a smart, funny, professional, thoughtful, and yet relaxed person - we are just two peas in a pod!
I also have a wonderful little boy named Jean-Marie who lives with his mother in Montréal, Québec, Canada, and whom I get to visit every few months in person and chat with him frequently via Skype. He loves both his mama and his papa very much despite them living in different cities - and we love him back! He is a sweet, funny, and kind human being - and he is cute, if I may say so in a fully biased manner (as clearly evidenced by the pictures below)! He is also a great excuse to visit the lovely city of Montréal to enjoy some French-Canadian hospitality and food! Vive la Québec!