INTER-RATER RELIABILITY · KRIPPENDORFF'S ALPHA · COHEN'S AND FLEISS' KAPPA · GWET'S AC1 · ICC · RUNS IN YOUR BROWSER · NOTHING UPLOADED

Do your raters agree beyond chance, and where do they not?

Paste the ratings two or more people gave the same things: coders labelling interview passages, annotators tagging text for a model, clinicians grading scans, judges scoring entries, teachers marking essays. You get Krippendorff's alpha at the right level of measurement, with missing ratings allowed and a bootstrap interval; Cohen's kappa for two raters, weighted when the categories are ordered; Fleiss' kappa and Gwet's AC1 for many; the six intraclass correlations for numeric scores; and the parts a single number hides: which category is confused, which rater pulls agreement down, and which items the raters split on.

Nothing you enter leaves this browser tab. Statistical output from data you supply, not advice. Not legal, tax, or financial advice.

Your ratings

Paste ratings or load the example.

Method and citation

Krippendorff's alpha. α = 1 − D_o / D_e: one minus observed disagreement over the disagreement expected by chance. Every pair of ratings of the same subject enters a coincidence matrix with weight 1/(m_u − 1), m_u that subject's number of ratings, so subjects with more raters do not count more; subjects with a single rating drop out; D_e comes from the pooled category totals. The difference function sets the level of measurement: nominal (any two different labels differ by 1), ordinal (by the number of ratings ranked between them), interval (squared difference) and ratio (squared difference over the squared sum) (Krippendorff 2004, 2011; Hayes & Krippendorff 2007). Its interval here resamples subjects with replacement and recomputes alpha each time, and the share of replicates below 0.667 and 0.800 is the probability Krippendorff recommends reporting.

Kappa and AC1. Cohen's kappa (1960) compares two raters' agreement with what their own category rates would give by chance; weighted kappa (Cohen 1968) gives partial credit to near misses, with linear or quadratic weights on the category ranks; its interval and z test use the large-sample standard errors of Fleiss, Cohen & Everitt (1969). Fleiss' kappa (1971) and Gwet's AC1 (2008), AC2 with weights, cover many raters; both are computed in Gwet's generalised form, which allows missing ratings and equals Fleiss' 1971 value on complete data. They share the observed agreement and differ in the chance term: Fleiss uses Σ π_k², AC1 uses Σ π_k (1 − π_k) / (q − 1), which stays small when one category dominates. Per-category kappa is Fleiss' kappa on the data recoded as "this category or not". With two raters the tool adds PABAK and, for two categories, the prevalence and bias indices of Byrt, Bishop & Carlin (1993); with three to fifteen raters, every pairwise kappa and their mean, Light's kappa (1971).

Intraclass correlations. For numeric ratings with the same raters on every subject, the two-way ANOVA mean squares give the six ICCs of Shrout & Fleiss (1979), named as in McGraw & Wong (1996), with their exact F-based intervals (Satterthwaite degrees of freedom for the absolute-agreement forms). Subjects missing a rating are left out of the ICC only, and the page says how many. The bands of Koo & Li (2016) are read off the interval.

Verified by tests in the source: alpha reproduces the four values published in Krippendorff's (2011) worked example (0.743, 0.815, 0.849, 0.797) and agrees with the krippendorff 0.9.0 package to 10⁻¹² for every level on data with missing ratings; Fleiss' and Gwet's coefficients, plain and quadratic-weighted, match irrCAC 0.4.4 to its five decimals and statsmodels' fleiss_kappa to 10⁻¹²; Cohen's kappa matches scikit-learn's cohen_kappa_score (none, linear, quadratic) and its standard errors match statsmodels 0.15.0's cohens_kappa to 10⁻¹²; the ICCs reproduce Shrout & Fleiss's (1979) table and match pingouin 0.7.0's intraclass_corr to 10⁻¹², with intervals equal to McGraw & Wong's formulas evaluated with scipy's F quantiles to 10⁻⁸. Wide, long and counts layouts of the same ratings give the same alpha, and relabelling categories or reordering raters and subjects leaves it unchanged.

BibTeX

@book{krippendorff2004content,
  author    = {Krippendorff, Klaus},
  title     = {Content Analysis: An Introduction to Its Methodology},
  edition   = {2}, publisher = {Sage}, address = {Thousand Oaks, CA}, year = {2004}
}
@misc{krippendorff2011computing,
  author = {Krippendorff, Klaus},
  title  = {Computing {K}rippendorff's Alpha-Reliability},
  howpublished = {Departmental Papers (ASC) 43, University of Pennsylvania},
  year   = {2011}, url = {https://repository.upenn.edu/asc_papers/43/}
}
@article{hayes2007answering,
  author  = {Hayes, Andrew F. and Krippendorff, Klaus},
  title   = {Answering the Call for a Standard Reliability Measure for Coding Data},
  journal = {Communication Methods and Measures},
  volume  = {1}, number = {1}, pages = {77--89}, year = {2007},
  doi     = {10.1080/19312450709336664}
}
@article{cohen1960coefficient,
  author  = {Cohen, Jacob},
  title   = {A Coefficient of Agreement for Nominal Scales},
  journal = {Educational and Psychological Measurement},
  volume  = {20}, number = {1}, pages = {37--46}, year = {1960},
  doi     = {10.1177/001316446002000104}
}
@article{cohen1968weighted,
  author  = {Cohen, Jacob},
  title   = {Weighted Kappa: Nominal Scale Agreement with Provision for Scaled Disagreement or Partial Credit},
  journal = {Psychological Bulletin},
  volume  = {70}, number = {4}, pages = {213--220}, year = {1968},
  doi     = {10.1037/h0026256}
}
@article{fleiss1969large,
  author  = {Fleiss, Joseph L. and Cohen, Jacob and Everitt, B. S.},
  title   = {Large Sample Standard Errors of Kappa and Weighted Kappa},
  journal = {Psychological Bulletin},
  volume  = {72}, number = {5}, pages = {323--327}, year = {1969},
  doi     = {10.1037/h0028106}
}
@article{fleiss1971measuring,
  author  = {Fleiss, Joseph L.},
  title   = {Measuring Nominal Scale Agreement Among Many Raters},
  journal = {Psychological Bulletin},
  volume  = {76}, number = {5}, pages = {378--382}, year = {1971},
  doi     = {10.1037/h0031619}
}
@article{gwet2008computing,
  author  = {Gwet, Kilem Li},
  title   = {Computing Inter-Rater Reliability and Its Variance in the Presence of High Agreement},
  journal = {British Journal of Mathematical and Statistical Psychology},
  volume  = {61}, number = {1}, pages = {29--48}, year = {2008},
  doi     = {10.1348/000711006X126600}
}
@article{shrout1979intraclass,
  author  = {Shrout, Patrick E. and Fleiss, Joseph L.},
  title   = {Intraclass Correlations: Uses in Assessing Rater Reliability},
  journal = {Psychological Bulletin},
  volume  = {86}, number = {2}, pages = {420--428}, year = {1979},
  doi     = {10.1037/0033-2909.86.2.420}
}
@article{mcgraw1996forming,
  author  = {McGraw, Kenneth O. and Wong, S. P.},
  title   = {Forming Inferences About Some Intraclass Correlation Coefficients},
  journal = {Psychological Methods},
  volume  = {1}, number = {1}, pages = {30--46}, year = {1996},
  doi     = {10.1037/1082-989X.1.1.30}
}
@article{landis1977measurement,
  author  = {Landis, J. Richard and Koch, Gary G.},
  title   = {The Measurement of Observer Agreement for Categorical Data},
  journal = {Biometrics},
  volume  = {33}, number = {1}, pages = {159--174}, year = {1977},
  doi     = {10.2307/2529310}
}
@article{koo2016guideline,
  author  = {Koo, Terry K. and Li, Mae Y.},
  title   = {A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research},
  journal = {Journal of Chiropractic Medicine},
  volume  = {15}, number = {2}, pages = {155--163}, year = {2016},
  doi     = {10.1016/j.jcm.2016.02.012}
}
@article{byrt1993bias,
  author  = {Byrt, Ted and Bishop, Janet and Carlin, John B.},
  title   = {Bias, Prevalence and Kappa},
  journal = {Journal of Clinical Epidemiology},
  volume  = {46}, number = {5}, pages = {423--429}, year = {1993},
  doi     = {10.1016/0895-4356(93)90018-V}
}
@article{light1971measures,
  author  = {Light, Richard J.},
  title   = {Measures of Response Agreement for Qualitative Data: Some Generalizations and Alternatives},
  journal = {Psychological Bulletin},
  volume  = {76}, number = {5}, pages = {365--377}, year = {1971},
  doi     = {10.1037/h0031643}
}

Limits

Questions

Which coefficient should I report?

For coded content, annotation or any data where raters vary from item to item or some ratings are missing, report Krippendorff's alpha at the level of measurement of your categories, with its interval. For exactly two raters who rated every item, Cohen's kappa is the familiar choice, weighted for ordered categories. For numeric scores by the same raters, report the intraclass correlation that matches your design, usually ICC(2,1), and say which form it is.

Why is my kappa low when the raters agree 90% of the time?

This is the kappa paradox. When one category dominates, two raters would agree often by chance alone, so kappa, which measures agreement beyond chance, can be low or even negative despite high raw agreement. Gwet's AC1 uses a chance term that is less affected by skewed categories. Report raw agreement, kappa and AC1 together and say how skewed the categories are.

What is a good Krippendorff's alpha?

Krippendorff recommends relying on data only where alpha is at least 0.800, and drawing tentative conclusions where it is between 0.667 and 0.800. Look at the lower end of the interval too: the tool reports how often the bootstrap fell below each standard. These are conventions for content analysis, not laws; a medical or safety decision may need more.

Which ICC is which?

ICC(1) treats every subject as rated by a different random set of raters. ICC(2) treats the raters as a random sample of possible raters and asks for absolute agreement, so a rater who scores everyone two points higher lowers it. ICC(3) treats these raters as the only ones of interest and asks only for consistency, so a constant offset does not count. The k forms are the reliability of the mean of all raters rather than of one.

What layouts can I paste?

Wide: one row per subject and one column per rater. Long: three columns, subject, rater and rating, as most annotation tools export. Counts: one row per subject and one column per category holding how many raters chose it. Empty cells, NA, a dot, a dash and a question mark count as missing.

Should I drop the rater whose removal raises alpha?

Not on that basis alone. Choosing raters to raise a reliability coefficient makes it look better than the coding process is. Use the table to find a coder who misread the instructions or a category that is badly defined, fix that, and code again, or report alpha both with and without the rater and say why.

Is my data uploaded anywhere?

No. The file is read and analysed by JavaScript in this browser tab. There are no accounts, cookies or analytics vendors.

Example data: 1,200 rows, one per rater per comment, for 300 Reddit comments from GoEmotions (Demszky et al. 2020, ACL; Apache License 2.0), each labelled by three to five of the dataset's raters. Each rater's emotion ticks are reduced to one sentiment with the dataset's own grouping (positive, negative, ambiguous, neutral, or mixed when a rater ticked emotions from more than one group); a rater who marked a comment "very unclear" or ticked nothing counts as missing. The comment text is not included. On these 300 comments alpha is 0.276: the raters agree on a comment's overall sentiment far less than a coding study would accept, most on "positive" and least on "mixed".