Skip to content

Evaluate

Evaluate pairwise agreement coefficient estimates for statistical significance

evaluate(result_a, result_b=None, metrics=None, alternative='two-sided', method='auto', n_permutations=1000, max_exact_permutations=10000, random_state=None)

Evaluate pairwise agreement coefficient estimates for statistical significance via permutation testing

Notes

Two-Sample Comparison (result_b provided):

  • \(H_0\): \(\text{coef}_A - \text{coef}_B = 0\) (no difference in agreement)
  • \(H_a\): \(\text{coef}_A - \text{coef}_B \neq 0\) (for alternative="two-sided")
  • Uses a swap-based null model (swapping item rows between datasets).

One-Sample Test (result_b=None):

  • \(H_0\): \(\text{coef}_A = 0\) (agreement is random chance)
  • \(H_a\): \(\text{coef}_A > 0\) or \(\neq 0\) (depending on alternative)
  • Uses a shuffle-based null model (shuffling labels within items).

Parameters:

Name Type Description Default
result_a AgreementResult

The primary agreement result object, derived from a call to compute()

 required
result_b AgreementResult | None

Optional agreement result object for two-sample comparison

 None
metrics list[str] | None

List of coefficient identifiers to compute, e.g., ["fleiss", "ac1"] If None, defaults to evaluating all metrics common in result objects

 None
alternative Alternative

Defines the direction of the alternative hypothesis \(H_a\) Options are "two-sided", "greater", "less"

 'two-sided'
method PermutationMethod

Defines the method of permutation to obtain \(p\)-values

Options are "exact", "monte_carlo", and "auto"

"auto" switches to "exact" if total possible permutations are below max_exact_permutations

 'auto'
n_permutations int

Number of Monte Carlo resamples to draw when method="monte_carlo"

 1000
max_exact_permutations int

Upper bound threshold for using exact enumerations when method="auto", i.e., obtaining the exact \(p\)-value

 10000
random_state int | None

Seed used to initialize random generator for reproducible resampling

 None

Returns:

Type Description
EvaluationResults

A container holding EvaluationResult objects for each tested metric, including point estimates, \(p\)-values, method details, and null distribution samples

Raises:

Type Description
ValueError
  • If either result object is missing its attached dataset
  • If result_a and result_b have non-matching item counts
  • If any requested metric is missing from the underlying result objects

Examples:

>>> # Two-sample test comparing two models
>>> eval_results = evaluate(
...     result_a=res_gpt4,
...     result_b=res_claude,
...     metrics=["cohen", "ac1"],
...     alternative="two-sided",
...     n_permutations=2000,
...     random_state=42
... )
>>> eval_results["cohen"].p_value
0.003
Source code in interrater/evaluate.py 
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
def evaluate(
    result_a: AgreementResult,
    result_b: AgreementResult | None = None,
    metrics: list[str] | None = None,
    alternative: Alternative = "two-sided",
    method: PermutationMethod = "auto",
    n_permutations: int = 1000,
    max_exact_permutations: int = 10000,
    random_state: int | None = None
) -> EvaluationResults:
    """
    Evaluate pairwise agreement coefficient estimates for statistical significance
    via permutation testing

    Notes
    -----
    **Two-Sample Comparison** (`result_b` provided):

    - $H_0$: $\\text{coef}_A - \\text{coef}_B = 0$ (no difference in agreement)
    - $H_a$: $\\text{coef}_A - \\text{coef}_B \\neq 0$ (for `alternative="two-sided"`)
    - Uses a *swap-based* null model (swapping item rows between datasets).

    **One-Sample Test** (`result_b=None`):

    - $H_0$: $\\text{coef}_A = 0$ (agreement is random chance)
    - $H_a$: $\\text{coef}_A > 0$ or $\\neq 0$ (depending on `alternative`)
    - Uses a *shuffle-based* null model (shuffling labels within items).

    Parameters
    ----------
    result_a: AgreementResult
        The primary agreement result object, derived from a call to `compute()`
    result_b: AgreementResult, default=None
        Optional agreement result object for two-sample comparison
    metrics: list[str], default=None
        List of coefficient identifiers to compute, e.g., ["fleiss", "ac1"]
        If None, defaults to evaluating all metrics common in result objects
    alternative: str, default="two-sided"
        Defines the direction of the alternative hypothesis $H_a$
        Options are "two-sided", "greater", "less"
    method: str, default="auto"
        Defines the method of permutation to obtain $p$-values

        Options are `"exact"`, `"monte_carlo"`, and `"auto"`

        `"auto"` switches to `"exact"` if total possible permutations are below 
        `max_exact_permutations`
    n_permutations: int, default=1000
        Number of Monte Carlo resamples to draw when `method="monte_carlo"`
    max_exact_permutations: int, default=10000
        Upper bound threshold for using exact enumerations when `method="auto"`,
        i.e., obtaining the exact $p$-value
    random_state: int, default=None
        Seed used to initialize random generator for reproducible resampling

    Returns
    -------
    EvaluationResults
        A container holding `EvaluationResult` objects for each tested metric, 
        including point estimates, $p$-values, method details, and null distribution samples

    Raises
    ------
    ValueError
        - If either result object is missing its attached `dataset`
        - If `result_a` and `result_b` have non-matching item counts
        - If any requested metric is missing from the underlying result objects

    Examples
    --------
    >>> # Two-sample test comparing two models
    >>> eval_results = evaluate(
    ...     result_a=res_gpt4,
    ...     result_b=res_claude,
    ...     metrics=["cohen", "ac1"],
    ...     alternative="two-sided",
    ...     n_permutations=2000,
    ...     random_state=42
    ... )
    >>> eval_results["cohen"].p_value
    0.003
    """
    if result_a.dataset is None:
        raise ValueError("AgreementResult A must contain a dataset to perform permutation testing")

    if result_b is not None and result_b.dataset is None:
        raise ValueError("AgreementResult B must contain a dataset to perform pairwise permutation testing")

    if result_b is not None:
        if result_a.n_items != result_b.n_items:
            raise ValueError(
                "Cannot compare agreement results with different numbers of items"
            )

    if result_b is None:
        metrics = metrics or result_a.metrics
    else:
        metrics = metrics or list(
            set(result_a.metrics) &
            set(result_b.metrics)
        )

    results = []

    for metric in metrics:
        if metric not in result_a.scores:
            raise ValueError(
                f"{metric} not found in first result. "
                f"Available: {list(result_a.scores.keys())}"
            )

        if result_b is not None and metric not in result_b.scores:
            raise ValueError(
                f"{metric} not found in second result. "
                f"Available: {list(result_b.scores.keys())}"
            )

        coefficient = result_a.coefficients[metric]

        if result_b is None:
            statistic = lambda ds: coefficient.compute(ds)
            null_model = "shuffle"
            datasets = (result_a.dataset,)
            estimate = result_a.scores[metric]
        else:
            statistic = lambda ds_a, ds_b: (coefficient.compute(ds_a) - coefficient.compute(ds_b))
            null_model = "swap"
            datasets = (result_a.dataset, result_b.dataset,)
            estimate = result_a.scores[metric] - result_b.scores[metric]

        test = PermutationTest(
            statistic=statistic,
            null_model=null_model,
            alternative=alternative,
            method=method,
            n_resamples=n_permutations,
            max_exact_permutations=max_exact_permutations,
            random_state=random_state
        )

        permutation_result = test.run(*datasets)

        results.append(
            EvaluationResult(
                metric=metric,
                estimate=estimate,
                p_value=permutation_result.p_value,
                null_distribution=permutation_result.null_distribution,
                alternative=alternative,
                method=permutation_result.method,
                label_a=result_a.models[0],
                label_b=result_b.models[0] if result_b else None
            )
        )

    return EvaluationResults(results)

:::