ITEM RESPONSE THEORY · RASCH / 1PL · 2PL · ITEM DIFFICULTY · DISCRIMINATION · S-X2 ITEM FIT · DISTRACTOR CHECK · TEST INFORMATION · RUNS IN YOUR BROWSER · NOTHING UPLOADED

Which questions on my test work, and how hard is each one really?

Quizzes, exams, certification tests, screening questionnaires: paste who got what right (or their raw A–E answers and the key). The tool fits item response theory models, the one-parameter (Rasch) and two-parameter logistic, by marginal maximum likelihood (Bock & Aitkin 1981). For every question it gives a difficulty and a discrimination with standard errors and a fit test, and a distractor table that shows when a wrong answer draws stronger students than the key.

Nothing you enter leaves this browser tab. A statistical estimate from the answers you give, not a substitute for a psychometrician's review of a high-stakes test.

Your answers

Example: the LSAT, and a key that was wrong

The real example is Section 6 of the Law School Admission Test: 1,000 examinees, 5 questions (Bock & Lieberman 1970), the standard first data set in IRT textbooks. The tool reproduces the estimates the R package ltm publishes for it (Rizopoulos 2006) to every printed digit: a common slope of 0.7551 (standard error 0.0694), question 1's difficulty −3.6153 (0.3266), and a log-likelihood of −2466.938. The 2PL adds four slopes but raises the log-likelihood by only 0.28 (likelihood-ratio chi-square 0.57 on 4 df, p = 0.967), so the 1PL is kept. All five questions are easy for this group; the test measures most precisely at ability −1.90, well below the average examinee, and its marginal reliability is 0.29.

The distractor table earns its keep on a second real test, SAT12 (600 grade-12 science students, 32 questions; shipped with the R package mirt, not with this page). Its documented key gives question 32 a discrimination of 0.12 and an item-rest correlation of 0.02. 44.3% chose answer 3, and they averaged 19.2 right on the other questions, against 18.5 for the 16.2% who chose the keyed answer 5. Rescored with 3 as the key, the question's discrimination rises to 0.47 and its item-rest correlation to 0.18. mirt's documentation reaches the same suspicion by a different model (the nominal response model): the key for question 32 "may be incorrect, and should be changed from 5 to 3".

Questions

What is item response theory?

Item response theory models the chance that a person gets a question right as a function of their ability and two properties of the question: its difficulty (the ability at which half of people get it right) and its discrimination (how sharply the chance rises around that point). Unlike a raw percent correct, the difficulty does not depend on how able this particular group happened to be, and each person's score comes with its own standard error.

What is the difference between the 1PL (Rasch) and 2PL models?

The 1PL gives every item the same discrimination, so the number right carries all the information about ability; this is the Rasch model. The 2PL lets each item discriminate differently, so a sharp question counts for more. The tool fits both, reports a likelihood-ratio test of one against the other, and keeps the one with the lower BIC unless you choose.

How do I find a miskeyed or bad question?

Give the raw answers (A to E or 1 to 5) and a row whose first cell is key. For each question the tool shows how many chose each answer and the average score on the rest of the test of the people who chose it. If a wrong answer is chosen by people who do better on the rest of the test than those who chose the key, the key or the wording deserves a look. A near-zero or negative discrimination points the same way.

How many people do I need?

The tool refuses fewer than 20 and warns below 100. A 1PL is usable with a few hundred people; a 2PL usually needs several hundred for stable discriminations. The standard errors say how precise each estimate is on your data, so read them before acting on a small difference.

What does S-X2 item fit mean?

S-X2 (Orlando and Thissen 2000) groups people by their total score and compares, for each group, the share who got the item right with the share the model predicts. A small p-value means the item does not behave as the model says, for example because guessing lifts the bottom of its curve or because it measures something else. It needs complete answers, so it is skipped when some are missing.

What does the tool not do?

It fits right/wrong items only, on one ability, with a normal ability distribution. It does not fit the three-parameter guessing model, partial-credit or graded items, several dimensions, or differential item functioning between groups, and it does not equate test forms. It is a statistical estimate, not a substitute for a psychometrician's review of a high-stakes test.

Is my data uploaded anywhere?

No. Every number is computed by JavaScript in this browser tab. There are no accounts, cookies or analytics vendors.

Method and citation

Models (Birnbaum 1968; Rasch 1960). P(right | ability θ) = 1 / (1 + exp(−a(θ − b))), on the logistic metric (no 1.7 scaling constant). The 1PL shares one slope a across questions with ability N(0, 1); in the Rasch metric that is slope 1 and an ability standard deviation of a.

Estimation (Bock & Aitkin 1981). Marginal maximum likelihood by EM over 61 quadrature points on [−6, 6] with normal weights: the E-step finds each response pattern's posterior over ability, the M-step solves each question's expected log-likelihood by Newton's method (the 1PL jointly, for its shared slope). Missing answers are treated as not presented. Standard errors come from the observed information, by central differences of the exact marginal score.

Item fit: S-X2 (Orlando & Thissen 2000), with the model's proportion right at each total score from the Lord & Wingersky (1984) recursion and neighbouring score groups pooled until both expected counts reach 1. Scores: EAP ability and posterior SD for every person (Bock & Mislevy 1982), and a number-right conversion table (Thissen et al. 1995). Test information: the sum of a²P(1 − P), whose inverse square root is the standard error of measurement at each ability. Marginal reliability: the variance of the EAP scores over that variance plus the mean posterior variance.

Checked against ltm's published LSAT output (Rizopoulos 2006: every estimate, standard error, log-likelihood, AIC, BIC and the likelihood-ratio test, to the printed digits), against the Python package girth on LSAT (within 0.0002) and on a simulated 15-question test (within 0.004), and on 60 simulated tests with a known truth: 95% intervals covered 95.1% of the true slopes and 95.5% of the true difficulties, and S-X2 flagged 5.1% of questions that fit at the 5% level. Implemented from the papers; no third-party code was ported.

@article{bock1981marginal, author={Bock, R. Darrell and Aitkin, Murray}, title={Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm}, journal={Psychometrika}, volume={46}, number={4}, pages={443--459}, year={1981}, doi={10.1007/BF02293801}}
@incollection{birnbaum1968some, author={Birnbaum, Allan}, title={Some latent trait models and their use in inferring an examinee's ability}, booktitle={Statistical Theories of Mental Test Scores}, editor={Lord, Frederic M. and Novick, Melvin R.}, publisher={Addison-Wesley}, pages={397--479}, year={1968}}
@book{rasch1960probabilistic, author={Rasch, Georg}, title={Probabilistic Models for Some Intelligence and Attainment Tests}, publisher={Danish Institute for Educational Research}, year={1960}}
@article{orlando2000likelihood, author={Orlando, Maria and Thissen, David}, title={Likelihood-based item-fit indices for dichotomous item response theory models}, journal={Applied Psychological Measurement}, volume={24}, number={1}, pages={50--64}, year={2000}, doi={10.1177/01466216000241003}}
@article{lord1984comparison, author={Lord, Frederic M. and Wingersky, Marilyn S.}, title={Comparison of IRT true-score and equipercentile observed-score "equatings"}, journal={Applied Psychological Measurement}, volume={8}, number={4}, pages={453--461}, year={1984}, doi={10.1177/014662168400800409}}
@article{bock1982adaptive, author={Bock, R. Darrell and Mislevy, Robert J.}, title={Adaptive EAP estimation of ability in a microcomputer environment}, journal={Applied Psychological Measurement}, volume={6}, number={4}, pages={431--444}, year={1982}, doi={10.1177/014662168200600405}}
@article{thissen1995item, author={Thissen, David and Pommerich, Mary and Billeaud, Kathleen and Williams, Valerie S. L.}, title={Item response theory for scores on tests including polytomous items with ordered responses}, journal={Applied Psychological Measurement}, volume={19}, number={1}, pages={39--49}, year={1995}, doi={10.1177/014662169501900105}}
@article{bock1970fitting, author={Bock, R. Darrell and Lieberman, Marcus}, title={Fitting a response model for n dichotomously scored items}, journal={Psychometrika}, volume={35}, number={2}, pages={179--197}, year={1970}, doi={10.1007/BF02291262}}
@article{rizopoulos2006ltm, author={Rizopoulos, Dimitris}, title={ltm: An R package for latent variable modeling and item response theory analyses}, journal={Journal of Statistical Software}, volume={17}, number={5}, pages={1--25}, year={2006}, doi={10.18637/jss.v017.i05}}

Example data: LSAT Section 6 response-pattern counts from Bock & Lieberman (1970), as distributed with the R package ltm. SAT12 is described from the R package mirt's documentation and is not redistributed here.

Limits: 20 to 20,000 people, 2 to 100 right/wrong questions, one ability dimension, normal ability distribution, no guessing parameter.