Using set coefficients¶

This notebook demonstrates how to compute interrater agreement with the set-based coefficients available in the package, organized by data level.

Note that the set-based coefficients currently only work with nominal data.

Set-valued metrics are designed for scenarios where a model's rating is a set of labels rather than a single discrete class (e.g., extracting multiple clinical findings for a patient, or assigning tags to a recommendation).

Supported coefficient definitions¶

Coefficient	General form
Jaccard similarity	$J(A, B) = \frac{\|A \cap B \|}{\|A \cup B\|}$ [1-2]
MASI similarity	$M(A, B) = J(A, B) \times m$ [3]

Note that MASI similarity extends the Jaccard similarity index by multiplying it by a monotonicity factor (weight) $m$ depending on the amount of overlap between the 2 sets, i.e., subset/superset relationships.

In [1]:

# Adding root directory for now before packaging

import sys
from pathlib import Path

sys.path.append(str(Path.cwd().parent.parent))

# Adding root directory for now before packaging import sys from pathlib import Path sys.path.append(str(Path.cwd().parent.parent))

In [2]:

# Imports

import pandas as pd

from interrater.base.dataset import Dataset
from interrater.preprocessing.levels_and_weights import LevelType
from interrater.compute import compute

# Imports import pandas as pd from interrater.base.dataset import Dataset from interrater.preprocessing.levels_and_weights import LevelType from interrater.compute import compute

In [3]:

# Mock dataset
# Note that sets should be tuples and not lists

set_df = pd.DataFrame([
    {"item": "Patient_A", "model": "GPT-4", "run": 0, "event": ("Missed, late, or insufficient prandial bolus", "Basal insufficiency", "Notable finding")},
    {"item": "Patient_A", "model": "Claude", "run": 0, "event": ("Missed, late, or insufficient prandial bolus", "Basal insufficiency")},  # Subset
    {"item": "Patient_A", "model": "Llama", "run": 0, "event": ("Missed, late, or insufficient prandial bolus", "Sensor artifact")},  # Partial overlap
    
    {"item": "Patient_B", "model": "GPT-4", "run": 0, "event": ("Overcorrection", "Basal excess")},
    {"item": "Patient_B", "model": "Claude", "run": 0, "event": ("Overcorrection", "Basal excess")},  # Perfect match
    {"item": "Patient_B", "model": "Llama", "run": 0, "event": ("Overcorrection", "Basal excess")},    # Perfect match
    
    {"item": "Patient_C", "model": "GPT-4", "run": 0, "event": ("Missed meal or incorrect prandial dose")},
    {"item": "Patient_C", "model": "Claude", "run": 0, "event": ("Ambiguous")},            # Disjoint
    {"item": "Patient_C", "model": "Llama", "run": 0, "event": ()}                       # Empty set
])

set_ds = Dataset(_df=set_df, target="event", level=LevelType.NOMINAL)
print(set_ds)

# Mock dataset # Note that sets should be tuples and not lists set_df = pd.DataFrame([ {"item": "Patient_A", "model": "GPT-4", "run": 0, "event": ("Missed, late, or insufficient prandial bolus", "Basal insufficiency", "Notable finding")}, {"item": "Patient_A", "model": "Claude", "run": 0, "event": ("Missed, late, or insufficient prandial bolus", "Basal insufficiency")}, # Subset {"item": "Patient_A", "model": "Llama", "run": 0, "event": ("Missed, late, or insufficient prandial bolus", "Sensor artifact")}, # Partial overlap {"item": "Patient_B", "model": "GPT-4", "run": 0, "event": ("Overcorrection", "Basal excess")}, {"item": "Patient_B", "model": "Claude", "run": 0, "event": ("Overcorrection", "Basal excess")}, # Perfect match {"item": "Patient_B", "model": "Llama", "run": 0, "event": ("Overcorrection", "Basal excess")}, # Perfect match {"item": "Patient_C", "model": "GPT-4", "run": 0, "event": ("Missed meal or incorrect prandial dose")}, {"item": "Patient_C", "model": "Claude", "run": 0, "event": ("Ambiguous")}, # Disjoint {"item": "Patient_C", "model": "Llama", "run": 0, "event": ()} # Empty set ]) set_ds = Dataset(_df=set_df, target="event", level=LevelType.NOMINAL) print(set_ds)

Dataset(n=9, target='event', level='LevelType.NOMINAL', models=3, items=3)

In [5]:

# Compute metrics

set_results = compute(
    df=set_ds.df,
    target="event",
    metrics=["jaccard", "masi"],
    level=LevelType.NOMINAL,
    ci=True,
    n_boot=1000
)

# Print results
set_results.__repr__

# Compute metrics set_results = compute( df=set_ds.df, target="event", metrics=["jaccard", "masi"], level=LevelType.NOMINAL, ci=True, n_boot=1000 ) # Print results set_results.__repr__

Out[5]:

<bound method AgreementResult.__repr__ of AgreementResult:
  jaccard: 0.3333 (SE=0.268, 95% CI=[0.000, 1.000])
  masi: 0.3333 (SE=0.269, 95% CI=[0.000, 1.000])>

In [6]:

# View as dataframe

set_results.to_dataframe()

# View as dataframe set_results.to_dataframe()

Out[6]:

	metric	score	standard_error	ci_lower	ci_upper	target	level	n_items	n_raters	models	dataset	metadata
0	jaccard	0.333333	0.267805	0.0	1.0	event	LevelType.NOMINAL	3	3	Claude,GPT-4,Llama	Dataset(n=9, target='event', level='LevelType....	{}
1	masi	0.333333	0.269479	0.0	1.0	event	LevelType.NOMINAL	3	3	Claude,GPT-4,Llama	Dataset(n=9, target='event', level='LevelType....	{}

References¶

1. Jaccard, P. (1901). Étude comparative de la distribution florale dans une portion des alpes et des jura. Bull Soc Vaudoise Sci Nat, 37:547–579.
2. Wikipedia. Jaccard index. (2021). https://en.wikipedia.org/wiki/Jaccard_index.
3. Passonneau, R. (2006). Measuring agreement on set-valued items (MASI) for semantic and pragmatic annotation.