Choosing coefficients
The choice of coefficient depends primarily on (1) the data level of measurement, and (2) the number of raters being compared.
The measurement levels in brief,
- Nominal: Data consists of items grouped into categories without any inherent order, hierarchy, or numerical value.
- Ordinal: Data consists of items that have an inherent order but where the intervals between them are unknown, unequal, or nonexistent.
- Interval and ratio: Data have inherent order with equal distances between data points. Interval data lacks an absolute, meaningful zero, whereas ratio data possesses it.
The level of measurement also determines the appropriate weighting method, if any, to use. Weights are scaling methods that define the penalty, or weight, that should be applied given a disagreement. The current weights supported are identity (nominal data), linear (ordinal data), and quadratic (interval and ratio data).
Certain coefficients only support measuring agreement between a certain number of raters. Cohen's \(\kappa\) and Scott's \(\pi\), for example, only support pairwise computation. When 2 or more raters exist, the coefficients that support multiple raters should be used, e.g., Fleiss' \(\kappa\) (multirater extension of Scott's \(\pi\)) or Gwet's \(\gamma\). See the table below for the supported number of raters for each coefficient in the package.
Implemented coefficients¶
The general form column shows the shared form between several of the chance-corrected coefficients1, where they differ only in the way they compute expected chance agreement \(p_e\). The associated \(p_e\) forms can be found in Using discrete coefficients. Note that observed agreement \(p_a\) for these coefficients is given by the general form of PercentAgreement, i.e., simple percent agreement does not correct for chance.
Although the general form is structurally the same as the other chance-corrected coefficients, Krippendorff's \(\alpha\) defines observed agreement differently to handle missing data, indicated by \(p_a'\) (definition is also in Using set coefficients).
| Coefficient class | Supported measurement levels | Supported number of raters | General form |
|---|---|---|---|
PercentAgreement |
Nominal | 2 | \(p_a = {\sum \sum}_{k, l} w_{kl} \pi_{kl}^{(k)}\) |
CohenKappa |
Nominal, Ordinal | 2 | \(\frac{p_a - p_e}{1 - p_e}\) |
ScottPi |
Nominal | 2 | \(\frac{p_a - p_e}{1 - p_e}\) |
FleissKappa |
Nominal | 2+ | \(\frac{p_a - p_e}{1 - p_e}\) |
GwetAC1 |
Nominal | 2+ | \(\frac{p_a - p_e}{1 - p_e}\) |
GwetAC2 |
Ordinal, Interval, Ratio | 2+ | \(\frac{p_a - p_e}{1 - p_e}\) |
KrippendorffAlpha |
Nominal, Ordinal, Interval, Ratio | 2+ | \(\frac{p'_a - p_e}{1 - p_e}\) |
KendallW |
Ordinal | 3+ | \(W_w = \frac{12S}{(n^3 - n)}\) |
JaccardSimilarity |
Set | 2 | \(J(A, B) = \frac{\|A \cap B \|}{\|A \cup B\|}\) |
MasiSimilarity |
Set | 2 | \(M(A, B) = J(A, B) \times m\) |
Coefficient tradeoffs¶
Chance-corrected coefficients¶
The tradeoffs for the chance-corrected coefficients are primarily based on how each one attempts to correct for chance. This section does not attempt to give a full account of the tradeoffs and paradoxes that occur when using these coefficients. We refer the interested reader to Gwet (2014), Zec et al. (2017), and Vach & Gerke (2023) for more.
Two paradoxes of Cohen's \(\kappa\) have been well-documented. Briefly, the prevalence paradox occurs when a rating is very common or very rare (i.e., highly prevalent). \(\kappa\) scores significantly drop regardless of interrater agreement, given that expected chance agreement \(p_e\) increases disproportionately. Second, the aysmmetry paradox occurs when the distribution of ratings (i.e., marginal totals) are asymmetrical such that imbalance between raters can inflate the \(\kappa\) score, regardless of lower overall concordance [7]. If the prevalence of categories in a categorical variable is relatively balanced, however, Cohen's \(\kappa\) often remains acceptable to use.
Several coefficients have been proposed to circumvent these paradoxes. Gwet's AC1, for example, aims to avoid the prevalence paradox with a different definition of expected chance agreement \(p_e\), namely, approximating based on the probability of raters assigning a rater completely at random instead of using the observed marginal probabilities. However, AC1 may yield high scores even when no actual agreement occurs between raters, and is not necessarily a direct substitute for Cohen's \(\kappa\) given its different definition that uses the probability of disagreement rather than agreement.
Krippendorff's \(\alpha\) also aims to avoid the prevalence paradox, and is a more generalized measure, handling varying numbers of raters, missing data, and all data levels. Although the coefficient handles any number of raters in principle, point estimates are biased downwards as the number of raters increase. Bootstrapping should thus be used to ensure accuracy.
Set coefficients¶
Set-based coefficients are designed for scenarios where a rating is a set of labels rather than a single discrete class, e.g., extracting multiple clinical findings for a patient, or assigning tags to a recommendation.
While Jaccard similarity is order-agnostic, e.g., {"A", "B"} and {"B", "A"} both yield a score of \(1.0\), it is strict in the sense that it disregards partial agreement. That is, it does not account for subset, superset, or overlap relationships. MASI similarity thus aims to consider these relationships by multiplying the Jaccard similarity index by a monotonicity factor depending on the amount of overlap between two sets. MASI is often used as the distance metric within Krippendorff's \(\alpha\), although this is not yet supported in the package. It should also be noted, however, that MASI is sensitive to cardinality. If set sizes are very different, MASI can penalize heavily even if overlap exists.
Interpreting coefficient estimates¶
The below table shows the score ranges for each of the coefficients, and the general heuristics commonly used to interpret them. These heuristics are general; specific acceptable thresholds depend on domain and context. Some of the chance-corrected coefficient heuristics (Cohen's \(\kappa\), Scott's \(\pi\), Fleiss' \(\kappa\), Gwet AC1/AC2, Kendall's \(W\)) use the widely adopted Landis & Koch (1977) benchmark, although again, this benchmark is general.
Note that for the coefficients with score range between \(-1\) and \(1\), a score of \(0.00\) represents agreement purely due to chance, and negative values represent agreement that occurs less than expected by chance.
| Coefficient | Score range | General heuristics |
|---|---|---|
PercentAgreement |
\([0, 1]\) |
|
CohenKappa |
\([-1, 1]\) |
|
ScottPi |
\([-1, 1]\) |
|
FleissKappa |
\([-1, 1]\) |
|
GwetAC1 |
\([-1, 1]\) |
|
GwetAC2 |
\([-1, 1]\) |
|
KrippendorffAlpha |
\([-1, 1]\) |
|
KendallW |
\([0, 1]\) |
|
JaccardSimilarity |
\([0, 1]\) |
|
MasiSimilarity |
\([0, 1]\) |
|
When intepreting coefficient estimates, it may also be necessary to evaluate whether the estimate is statistically significant. See Statistics for more information on methods used in the package.
Finally, consider the differences between reliability and validity. Agreement is reliable when raters consistently classify items into the same categories, and valid only if raters consistently classify items into the correct category, e.g., based on some gold-standard/ground-truth dataset [1]. In statistical terms, validity is equal to precision plus unbiasedness.
References¶
- Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters. Advanced Analytics, LLC.
- Kendall, M. G., & Gibbons, J. D. (1990). Rank correlation methods. New York, NY : Oxford University Press.
- Krippendorff, K. (2013). Content analysis: An introduction to its methodology, 3rd edition. Thousand Oaks, CA: Sage.
- Jaccard, P. (1901). Étude comparative de la distribution florale dans une portion des alpes et des jura. Bull Soc Vaudoise Sci Nat, 37:547–579.
- Wikipedia. Jaccard index, 2021. https://en.wikipedia.org/wiki/Jaccard_index.
- Passonneau, R. (2006). Measuring agreement on set-valued items (MASI) for semantic and pragmatic annotation.
- Zec, S., Soriani, N., Comoretto, R., & Baldi, I. (2017). High Agreement and High Prevalence: The Paradox of Cohen's Kappa. The open nursing journal, 11, 211–218. https://doi.org/10.2174/1874434601711010211
- Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.
- Marzi, G., Balzano, M., & Marchiori, D. (2024). K-Alpha calculator–krippendorff's alpha calculator: a user-friendly tool for computing krippendorff's alpha inter-rater reliability coefficient. MethodsX, 12, 102545.
- Vach, W., & Gerke, O. (2023). Gwet's AC1 is not a substitute for Cohen's kappa - A comparison of basic properties. MethodsX, 10, 102212. https://doi.org/10.1016/j.mex.2023.102212