Using discrete coefficients¶

This notebook demonstrates how to compute interrater agreement with the coefficients available for discrete data in the package, organized by data level. The formulas for each of the coefficients are as follows.

Supported coefficient definitions¶

Coefficient	General form	Expected chance agreement ($p_e$)
Cohen's $\kappa$	$\frac{p_a - p_e}{1 - p_e}$	$p_e = \sum_{k=1}^q p_k (p_{k+} p_{+k}^* + p_{+k}p_{k+}^*) / 2$ [1]
Scott's $\pi$	$\frac{p_a - p_e}{1 - p_e}$	$p_e = \sum_{k=1}^q \pi_k \pi_k^* p_k$ [1]
Fleiss's $\kappa$	$\frac{p_a - p_e}{1 - p_e}$	$p_e = \sum_{k, l}^q w_{kl} \pi_{k} \pi_{l}$ [1]
Gwet's AC1	$\frac{p_a - p_e}{1 - p_e}$	$p_e = \frac{1}{q-1} \sum_{k=1}^q \pi_k(1 - \pi_k)$ [1]
Gwet's AC2	$\frac{p_a - p_e}{1 - p_e}$	$p_e = \frac{u}{q(q-1)} \sum_{k=1}^q \pi_k(1 - \pi_k)$ [1]
Krippendorff's $\alpha$	$\frac{p'_a - p_e}{1 - p_e}$	$p_e = \hat{\pi}_1^2 + (1 - \hat{\pi}_1)^2$
Kendall's W (weighted with tie correction)	$W_w = \frac{12S}{(n^3 - n)}$ [2]	N/A
Percent agreement	$p_a = {\sum \sum}_{k, l} w_{kl} \pi_{kl}^{(k)}$ [1]	N/A

Note that for Krippendorff's $\alpha$, $p'_a = (1 - \epsilon_n) p_a + \epsilon_n$ [1, 3].

Setup¶

In [1]:

Copied!

# Adding root directory for now before packaging

import sys
from pathlib import Path

sys.path.append(str(Path.cwd().parent.parent))
# Adding root directory for now before packaging

import sys
from pathlib import Path

sys.path.append(str(Path.cwd().parent.parent))

In [2]:

Copied!

# Imports

import pandas as pd

from interrater.base.dataset import Dataset
from interrater.preprocessing.levels_and_weights import LevelType
from interrater.compute import compute
# Imports

import pandas as pd

from interrater.base.dataset import Dataset
from interrater.preprocessing.levels_and_weights import LevelType
from interrater.compute import compute

Nominal¶

Nominal data consists of items grouped into categories without any inherent order, hierarchy, or numerical value.

Supported coefficients¶

Cohen's $\kappa$ for 2 raters
Fleiss's $\kappa$
Scott's $\pi$
Gwet's AC1
Krippendorff's $\alpha$
Percent agreement

In [4]:

Copied!





# Mock data

nominal_df = pd.DataFrame([
    {"item": "Patient_A", "model": "GPT-4", "run": 0, "recommendations": "Prevent nocturnal hypoglycemia"},
    {"item": "Patient_A", "model": "Claude", "run": 0, "recommendations": "Prevent nocturnal hypoglycemia"},
    {"item": "Patient_A", "model": "Llama", "run": 0, "recommendations": "Improve prandial insulin timing"},
    
    {"item": "Patient_B", "model": "GPT-4", "run": 0, "recommendations": "Prevent prolonged hyperglycemia"},
    {"item": "Patient_B", "model": "Claude", "run": 0, "recommendations": "Increase insulin to carb ratio"},
    {"item": "Patient_B", "model": "Llama", "run": 0, "recommendations": "Increase basal insulin dos"},
    
    {"item": "Patient_C", "model": "GPT-4", "run": 0, "recommendations": "Carb counting"},
    {"item": "Patient_C", "model": "Claude", "run": 0, "recommendations": "Carb counting"},
    {"item": "Patient_C", "model": "Llama", "run": 0, "recommendations": "Carb counting"},
    
    {"item": "Patient_D", "model": "GPT-4", "run": 0, "recommendations": "Monitor ketones"},
    {"item": "Patient_D", "model": "Claude", "run": 0, "recommendations": "Prevent daytime hypoglycemia"},
    {"item": "Patient_D", "model": "Llama", "run": 0, "recommendations": "Improve prandial insulin adherence"}
])

nominal_ds = Dataset(_df=nominal_df, target="recommendations", level=LevelType.NOMINAL)
print(nominal_ds)
# Mock data

nominal_df = pd.DataFrame([
    {"item": "Patient_A", "model": "GPT-4", "run": 0, "recommendations": "Prevent nocturnal hypoglycemia"},
    {"item": "Patient_A", "model": "Claude", "run": 0, "recommendations": "Prevent nocturnal hypoglycemia"},
    {"item": "Patient_A", "model": "Llama", "run": 0, "recommendations": "Improve prandial insulin timing"},
    
    {"item": "Patient_B", "model": "GPT-4", "run": 0, "recommendations": "Prevent prolonged hyperglycemia"},
    {"item": "Patient_B", "model": "Claude", "run": 0, "recommendations": "Increase insulin to carb ratio"},
    {"item": "Patient_B", "model": "Llama", "run": 0, "recommendations": "Increase basal insulin dos"},
    
    {"item": "Patient_C", "model": "GPT-4", "run": 0, "recommendations": "Carb counting"},
    {"item": "Patient_C", "model": "Claude", "run": 0, "recommendations": "Carb counting"},
    {"item": "Patient_C", "model": "Llama", "run": 0, "recommendations": "Carb counting"},
    
    {"item": "Patient_D", "model": "GPT-4", "run": 0, "recommendations": "Monitor ketones"},
    {"item": "Patient_D", "model": "Claude", "run": 0, "recommendations": "Prevent daytime hypoglycemia"},
    {"item": "Patient_D", "model": "Llama", "run": 0, "recommendations": "Improve prandial insulin adherence"}
])

nominal_ds = Dataset(_df=nominal_df, target="recommendations", level=LevelType.NOMINAL)
print(nominal_ds)

Dataset(n=12, target='recommendations', level='LevelType.NOMINAL', models=3, items=4)

In [5]:

Copied!





# Compute Cohen's kappa for 2 models

pairwise_nominal_ds = nominal_ds.slice(model=["GPT-4", "Claude"])

cohen_result = compute(
    df=pairwise_nominal_ds.df,
    target="recommendations",
    metrics=["cohen"],
    level=LevelType.NOMINAL
)

# Print
print(f"{'Cohen (GPT-4 vs Claude)'}: {cohen_result.__repr__}")
# Compute Cohen's kappa for 2 models

pairwise_nominal_ds = nominal_ds.slice(model=["GPT-4", "Claude"])

cohen_result = compute(
    df=pairwise_nominal_ds.df,
    target="recommendations",
    metrics=["cohen"],
    level=LevelType.NOMINAL
)

# Print
print(f"{'Cohen (GPT-4 vs Claude)'}: {cohen_result.__repr__}")

Cohen (GPT-4 vs Claude): <bound method AgreementResult.__repr__ of AgreementResult:
  cohen: 0.4286>

In [6]:

Copied!





# Compute multirater metrics

multirater_results = compute(
    df=nominal_ds.df,
    target="recommendations",
    metrics=["percent_agreement", "fleiss", "scott", "ac1", "krippendorff"],
    level=LevelType.NOMINAL
)

# Print
multirater_results.__repr__
# Compute multirater metrics

multirater_results = compute(
    df=nominal_ds.df,
    target="recommendations",
    metrics=["percent_agreement", "fleiss", "scott", "ac1", "krippendorff"],
    level=LevelType.NOMINAL
)

# Print
multirater_results.__repr__

Out[6]:

<bound method AgreementResult.__repr__ of AgreementResult:
  percent_agreement: 0.3333
  fleiss: 0.2258
  scott: 0.2258
  ac1: 0.2529
  krippendorff: 0.2903>

In [6]:

Copied!

# View result metrics as a dict

multirater_results.to_dict()
# View result metrics as a dict

multirater_results.to_dict()

Out[6]:

{'percent_agreement': 0.3333333333333333,
 'fleiss': np.float64(0.2258064516129032),
 'scott': 0.2258064516129032,
 'ac1': np.float64(0.2529182879377431),
 'krippendorff': 0.29032258064516137}

In [7]:

Copied!

# View full result objects as a dataframe

multirater_results.to_dataframe()
# View full result objects as a dataframe

multirater_results.to_dataframe()

Out[7]:

	metric	score	standard_error	ci_lower	ci_upper	target	level	n_items	n_raters	models	dataset	metadata
0	percent_agreement	0.333333	NaN	NaN	NaN	recommendations	LevelType.NOMINAL	4	3	Claude,GPT-4,Llama	Dataset(n=12, target='recommendations', level=...	{}
1	fleiss	0.225806	NaN	NaN	NaN	recommendations	LevelType.NOMINAL	4	3	Claude,GPT-4,Llama	Dataset(n=12, target='recommendations', level=...	{}
2	scott	0.225806	NaN	NaN	NaN	recommendations	LevelType.NOMINAL	4	3	Claude,GPT-4,Llama	Dataset(n=12, target='recommendations', level=...	{}
3	ac1	0.252918	NaN	NaN	NaN	recommendations	LevelType.NOMINAL	4	3	Claude,GPT-4,Llama	Dataset(n=12, target='recommendations', level=...	{}
4	krippendorff	0.290323	NaN	NaN	NaN	recommendations	LevelType.NOMINAL	4	3	Claude,GPT-4,Llama	Dataset(n=12, target='recommendations', level=...	{}

Ordinal¶

Ordinal data consists of items that have an inherent order but where the intervals between them are unknown, unequal, or nonexistent.

Supported coefficients¶

Kendall's W
Gwet's AC2 with linear distance weights
Krippendorff's $\alpha$ (ordinal)
Weighted Cohen's $\kappa$ for 2 raters

In [8]:

Copied!





# Mock dataset

ordinal_df = pd.DataFrame([
    {"item": "finding_a", "model": "GPT-4", "run": 0, "rank": 1},
    {"item": "finding_a", "model": "Claude", "run": 0, "rank": 2},
    {"item": "finding_a", "model": "Llama", "run": 0, "rank": 1},
    
    {"item": "finding_b", "model": "GPT-4", "run": 0, "rank": 2},
    {"item": "finding_b", "model": "Claude", "run": 0, "rank": 3},
    {"item": "finding_b", "model": "Llama", "run": 0, "rank": 2},
    
    {"item": "finding_c", "model": "GPT-4", "run": 0, "rank": 3},
    {"item": "finding_c", "model": "Claude", "run": 0, "rank": 1},
    {"item": "finding_c", "model": "Llama", "run": 0, "rank": 2},
    
    {"item": "finding_d", "model": "GPT-4", "run": 0, "rank": 4},
    {"item": "finding_d", "model": "Claude", "run": 0, "rank": 4},
    {"item": "finding_d", "model": "Llama", "run": 0, "rank": 4}
])

ordinal_ds = Dataset(_df=ordinal_df, target="rank", level=LevelType.ORDINAL)
print(ordinal_ds)
# Mock dataset

ordinal_df = pd.DataFrame([
    {"item": "finding_a", "model": "GPT-4", "run": 0, "rank": 1},
    {"item": "finding_a", "model": "Claude", "run": 0, "rank": 2},
    {"item": "finding_a", "model": "Llama", "run": 0, "rank": 1},
    
    {"item": "finding_b", "model": "GPT-4", "run": 0, "rank": 2},
    {"item": "finding_b", "model": "Claude", "run": 0, "rank": 3},
    {"item": "finding_b", "model": "Llama", "run": 0, "rank": 2},
    
    {"item": "finding_c", "model": "GPT-4", "run": 0, "rank": 3},
    {"item": "finding_c", "model": "Claude", "run": 0, "rank": 1},
    {"item": "finding_c", "model": "Llama", "run": 0, "rank": 2},
    
    {"item": "finding_d", "model": "GPT-4", "run": 0, "rank": 4},
    {"item": "finding_d", "model": "Claude", "run": 0, "rank": 4},
    {"item": "finding_d", "model": "Llama", "run": 0, "rank": 4}
])

ordinal_ds = Dataset(_df=ordinal_df, target="rank", level=LevelType.ORDINAL)
print(ordinal_ds)

Dataset(n=12, target='rank', level='LevelType.ORDINAL', models=3, items=4)

In [9]:

Copied!





# Compute weighted Cohen's kappa for 2 models

pairwise_ordinal_ds = ordinal_ds.slice(model=["GPT-4", "Llama"])

cohen_weighted_result = compute(
    df=pairwise_ordinal_ds.df,
    target="rank",
    metrics=["cohen"],
    level=LevelType.ORDINAL
)

print(f"{'Cohen (Weighted)'}: {cohen_weighted_result.__repr__}")
# Compute weighted Cohen's kappa for 2 models

pairwise_ordinal_ds = ordinal_ds.slice(model=["GPT-4", "Llama"])

cohen_weighted_result = compute(
    df=pairwise_ordinal_ds.df,
    target="rank",
    metrics=["cohen"],
    level=LevelType.ORDINAL
)

print(f"{'Cohen (Weighted)'}: {cohen_weighted_result.__repr__}")

Cohen (Weighted): <bound method AgreementResult.__repr__ of AgreementResult:
  cohen: 0.6667>

In [10]:

Copied!





# Compute multirater metrics

ordinal_results = compute(
    df=ordinal_ds.df,
    target="rank",
    metrics=["kendall", "ac2", "krippendorff"],
    level=LevelType.ORDINAL
)

ordinal_results.__repr__
# Compute multirater metrics

ordinal_results = compute(
    df=ordinal_ds.df,
    target="rank",
    metrics=["kendall", "ac2", "krippendorff"],
    level=LevelType.ORDINAL
)

ordinal_results.__repr__

Out[10]:

<bound method AgreementResult.__repr__ of AgreementResult:
  kendall: 0.0889
  ac2: 0.4801
  krippendorff: 0.6361>

In [11]:

Copied!

ordinal_results.to_dataframe()
ordinal_results.to_dataframe()

Out[11]:

	metric	score	standard_error	ci_lower	ci_upper	target	level	n_items	n_raters	models	dataset	metadata
0	kendall	0.088889	NaN	NaN	NaN	rank	LevelType.ORDINAL	4	3	Claude,GPT-4,Llama	Dataset(n=12, target='rank', level='LevelType....	{}
1	ac2	0.480144	NaN	NaN	NaN	rank	LevelType.ORDINAL	4	3	Claude,GPT-4,Llama	Dataset(n=12, target='rank', level='LevelType....	{}
2	krippendorff	0.636080	NaN	NaN	NaN	rank	LevelType.ORDINAL	4	3	Claude,GPT-4,Llama	Dataset(n=12, target='rank', level='LevelType....	{}

Interval and ratio¶

Interval and ratio data both have inherent order with equal distances between data points. Interval data lacks an absolute, meaningful zero, whereas ratio data possesses it.

Supported coefficients¶

Gwet's AC2 with quadratic weights
Krippendorff's $\alpha$ (interval/ratio)

In [12]:

Copied!





# Mock dataset

interval_df = pd.DataFrame([
    {"item": "Task_1", "model": "GPT-4", "run": 0, "score": 10},
    {"item": "Task_1", "model": "Claude", "run": 0, "score": 11},
    {"item": "Task_1", "model": "Llama", "run": 0, "score": 10},
    
    {"item": "Task_2", "model": "GPT-4", "run": 0, "score": 20},
    {"item": "Task_2", "model": "Claude", "run": 0, "score": 22},
    {"item": "Task_2", "model": "Llama", "run": 0, "score": 19},
    
    {"item": "Task_3", "model": "GPT-4", "run": 0, "score": 15},
    {"item": "Task_3", "model": "Claude", "run": 0, "score": 14},
    {"item": "Task_3", "model": "Llama", "run": 0, "score": 15},
    
    {"item": "Task_4", "model": "GPT-4", "run": 0, "score": 5},
    {"item": "Task_4", "model": "Claude", "run": 0, "score": 8},
    {"item": "Task_4", "model": "Llama", "run": 0, "score": 6}
])

interval_ds = Dataset(_df=interval_df, target="score", level=LevelType.INTERVAL)
print(interval_ds)
# Mock dataset

interval_df = pd.DataFrame([
    {"item": "Task_1", "model": "GPT-4", "run": 0, "score": 10},
    {"item": "Task_1", "model": "Claude", "run": 0, "score": 11},
    {"item": "Task_1", "model": "Llama", "run": 0, "score": 10},
    
    {"item": "Task_2", "model": "GPT-4", "run": 0, "score": 20},
    {"item": "Task_2", "model": "Claude", "run": 0, "score": 22},
    {"item": "Task_2", "model": "Llama", "run": 0, "score": 19},
    
    {"item": "Task_3", "model": "GPT-4", "run": 0, "score": 15},
    {"item": "Task_3", "model": "Claude", "run": 0, "score": 14},
    {"item": "Task_3", "model": "Llama", "run": 0, "score": 15},
    
    {"item": "Task_4", "model": "GPT-4", "run": 0, "score": 5},
    {"item": "Task_4", "model": "Claude", "run": 0, "score": 8},
    {"item": "Task_4", "model": "Llama", "run": 0, "score": 6}
])

interval_ds = Dataset(_df=interval_df, target="score", level=LevelType.INTERVAL)
print(interval_ds)

Dataset(n=12, target='score', level='LevelType.INTERVAL', models=3, items=4)

In [13]:

Copied!





# Compute metrics

interval_results = compute(
    df=interval_ds.df,
    target="score",
    metrics=["ac2", "krippendorff"],
    level=LevelType.INTERVAL
)

interval_results.__repr__
# Compute metrics

interval_results = compute(
    df=interval_ds.df,
    target="score",
    metrics=["ac2", "krippendorff"],
    level=LevelType.INTERVAL
)

interval_results.__repr__

Out[13]:

<bound method AgreementResult.__repr__ of AgreementResult:
  ac2: 0.7033
  krippendorff: 0.9157>

In [14]:

Copied!

interval_results.to_dataframe()
interval_results.to_dataframe()

Out[14]:

	metric	score	standard_error	ci_lower	ci_upper	target	level	n_items	n_raters	models	dataset	metadata
0	ac2	0.703297	NaN	NaN	NaN	score	LevelType.INTERVAL	4	3	Claude,GPT-4,Llama	Dataset(n=12, target='score', level='LevelType...	{}
1	krippendorff	0.915709	NaN	NaN	NaN	score	LevelType.INTERVAL	4	3	Claude,GPT-4,Llama	Dataset(n=12, target='score', level='LevelType...	{}

References¶

Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters. Advanced Analytics, LLC.
Kendall, M. G., & Gibbons, J. D. (1990). Rank correlation methods. New York, NY : Oxford University Press.
Krippendorff, K. (2013). Content analysis: An introduction to its methodology, 3rd edition. Thousand Oaks, CA: Sage.