Skip to content

Results

Result containers for coefficient estimates, multi-field evaluations, bootstrap uncertainty estimates, and hypothesis testing via permutation.

AgreementResult dataclass

Immutable container holding agreement coefficient estimates and metadata.

Return object of interrater.compute(). Stores point estimates, optional bootstrap standard errors and BCa confidence intervals.

Parameters:

Name Type Description Default
scores dict[str, float]

Map of metric names to point estimate scores (e.g., {"cohen": 0.74}).

 required
dataset Dataset

Underlying Dataset instance.

 required
target str

Name of the target annotation column.

 required
level str

Level of measurement.

 required
n_items int

Total number of unique items evaluated.

 required
n_raters int

Total number of unique raters (model, run) pairs in the dataset.

 required
coefficients dict[str, AgreementCoefficient]

Map of metric names to instantiated AgreementCoefficient objects.

 required
models list[str]

List of unique model names included in the evaluation dataset.

 []
runs list[int]

List of unique run identifiers included in the evaluation dataset.

 []
standard_errors dict[str, float]

Map of metric names to bootstrap standard error estimates.

 {}
ci_ranges dict[str, tuple[float, float]]

Map of metric names to (lower_bound, upper_bound) BCa confidence intervals.

 {}
metadata dict[str, Any]

Arbitrary key-value user metadata attached to the result.

 {}

Attributes:

Name Type Description
metrics list[str]

List of calculated metric names present in the result.

Examples:

>>> result = compute(df, target="rating", metrics=["cohen", "ac1"], ci=True)
>>> result.score("cohen")
0.742
>>> result.to_dataframe()
     metric  score  standard_error  ci_lower  ci_upper ...
0     cohen  0.742           0.031     0.681     0.803
1       ac1  0.810           0.024     0.762     0.858
Source code in interrater/base/result.py 
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
@dataclass(slots=True, frozen=True)
class AgreementResult:
    """
    Immutable container holding agreement coefficient estimates and metadata.

    Return object of `interrater.compute()`. Stores point estimates,
    optional bootstrap standard errors and BCa confidence intervals.

    Parameters
    ----------
    scores : dict[str, float]
        Map of metric names to point estimate scores (e.g., `{"cohen": 0.74}`).
    dataset : Dataset
        Underlying `Dataset` instance.
    target : str
        Name of the target annotation column.
    level : str
        Level of measurement.
    n_items : int
        Total number of unique items evaluated.
    n_raters : int
        Total number of unique raters `(model, run)` pairs in the dataset.
    coefficients : dict[str, AgreementCoefficient]
        Map of metric names to instantiated `AgreementCoefficient` objects.
    models : list[str], default=[]
        List of unique model names included in the evaluation dataset.
    runs : list[int], default=[]
        List of unique run identifiers included in the evaluation dataset.
    standard_errors : dict[str, float], default={}
        Map of metric names to bootstrap standard error estimates.
    ci_ranges : dict[str, tuple[float, float]], default={}
        Map of metric names to `(lower_bound, upper_bound)` BCa confidence intervals.
    metadata : dict[str, Any], default={}
        Arbitrary key-value user metadata attached to the result.

    Attributes
    ----------
    metrics : list[str]
        List of calculated metric names present in the result.

    Examples
    --------
    >>> result = compute(df, target="rating", metrics=["cohen", "ac1"], ci=True)
    >>> result.score("cohen")
    0.742
    >>> result.to_dataframe()
         metric  score  standard_error  ci_lower  ci_upper ...
    0     cohen  0.742           0.031     0.681     0.803
    1       ac1  0.810           0.024     0.762     0.858
    """
    scores: dict[str, float]
    dataset: Dataset
    target: str
    level: str
    n_items: int
    n_raters: int
    coefficients: dict[str, AgreementCoefficient]
    models: list[str] = field(default_factory=list)
    runs: list[int] = field(default_factory=list)
    # optional statistics
    standard_errors: dict[str, float] = field(default_factory=dict)
    ci_ranges: dict[str, tuple[float, float]] = field(default_factory=dict)
    metadata: dict[str, Any] = field(default_factory=dict)

    @property
    def metrics(self) -> list[str]:
        """Return a list of metric names present in this result."""
        return list(self.scores.keys())

    def score(self, metric: str) -> float:
        """
        Retrieve the point estimate score for a specific metric.

        Parameters
        ----------
        metric : str
            Name of the metric.

        Returns
        -------
        float
            Point estimate score.

        Raises
        ------
        ValueError
            If the requested metric was not evaluated in this result.

        Examples
        --------
        >>> from interrater.compute import compute
        >>> res_gpt5 = compute(
        ...     df=rec_gpt5_dataset.df, 
        ...     target="rating",
        ...     metrics=["fleiss", "percent_agreement"],
        ...     ci=True,
        ...     n_boot=1000
        ... )
        >>> res_gpt5.metrics
        ['fleiss', 'percent_agreement']
        """
        if metric not in self.scores:
            raise ValueError(
                f"Metric {metric} not found."
                f"Available: {list(self.scores)}"
            )

        return self.scores[metric]

    def to_dict(self) -> dict[str, float]:
        """
        Return a simple dictionary mapping metric names to point estimates.

        Returns
        -------
        dict[str, float]
            Map of metric names to floating-point scores.
        """
        return dict(self.scores)

    def to_dataframe(self) -> pd.DataFrame:
        """
        Convert results into a tidy pandas DataFrame.

        Returns
        -------
        pd.DataFrame
            DataFrame with columns for metric, score, standard_error, ci_lower,
            ci_upper, target, level, n_items, n_raters, models, dataset, and metadata.

        Examples
        --------
        >>> from interrater.compute import compute
        >>> res_gpt5 = compute(
        ...     df=rec_gpt5_dataset.df, 
        ...     target="rating",
        ...     metrics=["fleiss", "percent_agreement"],
        ...     ci=True,
        ...     n_boot=1000
        ... )
        >>> res_gpt5.to_dict()
        {'fleiss': np.float64(0.6923076923076924),
        'percent_agreement': 0.8333333333333334}
        """
        records = []
        for metric, score in self.scores.items():
            ci = self.ci_ranges.get(metric, (float("nan"), float("nan")))
            records.append({
                "metric": metric,
                "score": score,
                "standard_error": self.standard_errors.get(metric, float("nan")),
                "ci_lower": ci[0],
                "ci_upper": ci[1],
                "target": self.target,
                "level": self.level,
                "n_items": self.n_items,
                "n_raters": self.n_raters,
                "models": ",".join(self.models),
                "dataset": self.dataset,
                "metadata": self.metadata
            })

        return pd.DataFrame(records)

    def select_metrics(self, metrics: list[str]) -> "AgreementResult":
        """
        Create a new `AgreementResult` restricted to a subset of metrics.

        Parameters
        ----------
        metrics : list[str]
            List of metric names to retain.

        Returns
        -------
        AgreementResult
            A new `AgreementResult` instance containing only specified metrics.

        Raises
        ------
        ValueError
            If none of the specified metrics exist in the current result.

        Examples
        --------
        >>> between = compute(
        ...     rec_dataset.df,
        ...     target="rating",
        ...     metrics=["fleiss", "percent_agreement"],
        ...     models=["gpt-5.4-mini", "gpt-5.4-nano"], # if using all models in the dataset, models doesn't need to be specified
        ...     ci=True,
        ...     n_boot=1000
        ... )
        >>> between.select_metrics(metrics=['fleiss'])
        """
        selected = [m for m in metrics if m in self.scores]

        if not selected:
            raise ValueError(
                f"None of the requested metrics are available. "
                f"Available: {self.metrics}"
            )

        return AgreementResult(
            scores={m: self.scores[m] for m in metrics},
            standard_errors={m: self.standard_errors[m] for m in metrics if m in self.standard_errors},
            ci_ranges={m: self.ci_ranges[m] for m in metrics if m in self.ci_ranges},
            coefficients={m: self.coefficients[m] for m in metrics},
            dataset=self.dataset,
            target=self.target,
            level=self.level,
            n_items=self.n_items,
            n_raters=self.n_raters,
            models=self.models,
            runs=self.runs,
            metadata=self.metadata
        )

    def __repr__(self) -> str:
        # Nicer printing for e.g., notebooks
        lines = ["AgreementResult:"]
        for metric, score in self.scores.items():
            line = f"  {metric}: {score:.4f}"
            if metric in self.standard_errors:
                se = self.standard_errors[metric]
                ci = self.ci_ranges[metric]
                line += f" (SE={se:.4f}, 95% CI=[{ci[0]:.4f}, {ci[1]:.4f}])"
            lines.append(line)
        return "\n".join(lines)

metrics property

Return a list of metric names present in this result.

score(metric)

Retrieve the point estimate score for a specific metric.

Parameters:

Name Type Description Default
metric str

Name of the metric.

 required

Returns:

Type Description
float

Point estimate score.

Raises:

Type Description
ValueError

If the requested metric was not evaluated in this result.

Examples:

>>> from interrater.compute import compute
>>> res_gpt5 = compute(
...     df=rec_gpt5_dataset.df, 
...     target="rating",
...     metrics=["fleiss", "percent_agreement"],
...     ci=True,
...     n_boot=1000
... )
>>> res_gpt5.metrics
['fleiss', 'percent_agreement']
Source code in interrater/base/result.py 
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
def score(self, metric: str) -> float:
    """
    Retrieve the point estimate score for a specific metric.

    Parameters
    ----------
    metric : str
        Name of the metric.

    Returns
    -------
    float
        Point estimate score.

    Raises
    ------
    ValueError
        If the requested metric was not evaluated in this result.

    Examples
    --------
    >>> from interrater.compute import compute
    >>> res_gpt5 = compute(
    ...     df=rec_gpt5_dataset.df, 
    ...     target="rating",
    ...     metrics=["fleiss", "percent_agreement"],
    ...     ci=True,
    ...     n_boot=1000
    ... )
    >>> res_gpt5.metrics
    ['fleiss', 'percent_agreement']
    """
    if metric not in self.scores:
        raise ValueError(
            f"Metric {metric} not found."
            f"Available: {list(self.scores)}"
        )

    return self.scores[metric]

to_dict()

Return a simple dictionary mapping metric names to point estimates.

Returns:

Type Description
dict[str, float]

Map of metric names to floating-point scores.
Source code in interrater/base/result.py 
127
128
129
130
131
132
133
134
135
136
def to_dict(self) -> dict[str, float]:
    """
    Return a simple dictionary mapping metric names to point estimates.

    Returns
    -------
    dict[str, float]
        Map of metric names to floating-point scores.
    """
    return dict(self.scores)

to_dataframe()

Convert results into a tidy pandas DataFrame.

Returns:

Type Description
DataFrame

DataFrame with columns for metric, score, standard_error, ci_lower, ci_upper, target, level, n_items, n_raters, models, dataset, and metadata.

Examples:

>>> from interrater.compute import compute
>>> res_gpt5 = compute(
...     df=rec_gpt5_dataset.df, 
...     target="rating",
...     metrics=["fleiss", "percent_agreement"],
...     ci=True,
...     n_boot=1000
... )
>>> res_gpt5.to_dict()
{'fleiss': np.float64(0.6923076923076924),
'percent_agreement': 0.8333333333333334}
Source code in interrater/base/result.py 
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
def to_dataframe(self) -> pd.DataFrame:
    """
    Convert results into a tidy pandas DataFrame.

    Returns
    -------
    pd.DataFrame
        DataFrame with columns for metric, score, standard_error, ci_lower,
        ci_upper, target, level, n_items, n_raters, models, dataset, and metadata.

    Examples
    --------
    >>> from interrater.compute import compute
    >>> res_gpt5 = compute(
    ...     df=rec_gpt5_dataset.df, 
    ...     target="rating",
    ...     metrics=["fleiss", "percent_agreement"],
    ...     ci=True,
    ...     n_boot=1000
    ... )
    >>> res_gpt5.to_dict()
    {'fleiss': np.float64(0.6923076923076924),
    'percent_agreement': 0.8333333333333334}
    """
    records = []
    for metric, score in self.scores.items():
        ci = self.ci_ranges.get(metric, (float("nan"), float("nan")))
        records.append({
            "metric": metric,
            "score": score,
            "standard_error": self.standard_errors.get(metric, float("nan")),
            "ci_lower": ci[0],
            "ci_upper": ci[1],
            "target": self.target,
            "level": self.level,
            "n_items": self.n_items,
            "n_raters": self.n_raters,
            "models": ",".join(self.models),
            "dataset": self.dataset,
            "metadata": self.metadata
        })

    return pd.DataFrame(records)

select_metrics(metrics)

Create a new AgreementResult restricted to a subset of metrics.

Parameters:

Name Type Description Default
metrics list[str]

List of metric names to retain.

 required

Returns:

Type Description
AgreementResult

A new AgreementResult instance containing only specified metrics.

Raises:

Type Description
ValueError

If none of the specified metrics exist in the current result.

Examples:

>>> between = compute(
...     rec_dataset.df,
...     target="rating",
...     metrics=["fleiss", "percent_agreement"],
...     models=["gpt-5.4-mini", "gpt-5.4-nano"], # if using all models in the dataset, models doesn't need to be specified
...     ci=True,
...     n_boot=1000
... )
>>> between.select_metrics(metrics=['fleiss'])
Source code in interrater/base/result.py 
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
def select_metrics(self, metrics: list[str]) -> "AgreementResult":
    """
    Create a new `AgreementResult` restricted to a subset of metrics.

    Parameters
    ----------
    metrics : list[str]
        List of metric names to retain.

    Returns
    -------
    AgreementResult
        A new `AgreementResult` instance containing only specified metrics.

    Raises
    ------
    ValueError
        If none of the specified metrics exist in the current result.

    Examples
    --------
    >>> between = compute(
    ...     rec_dataset.df,
    ...     target="rating",
    ...     metrics=["fleiss", "percent_agreement"],
    ...     models=["gpt-5.4-mini", "gpt-5.4-nano"], # if using all models in the dataset, models doesn't need to be specified
    ...     ci=True,
    ...     n_boot=1000
    ... )
    >>> between.select_metrics(metrics=['fleiss'])
    """
    selected = [m for m in metrics if m in self.scores]

    if not selected:
        raise ValueError(
            f"None of the requested metrics are available. "
            f"Available: {self.metrics}"
        )

    return AgreementResult(
        scores={m: self.scores[m] for m in metrics},
        standard_errors={m: self.standard_errors[m] for m in metrics if m in self.standard_errors},
        ci_ranges={m: self.ci_ranges[m] for m in metrics if m in self.ci_ranges},
        coefficients={m: self.coefficients[m] for m in metrics},
        dataset=self.dataset,
        target=self.target,
        level=self.level,
        n_items=self.n_items,
        n_raters=self.n_raters,
        models=self.models,
        runs=self.runs,
        metadata=self.metadata
    )

MultiAgreementResult dataclass

Immutable container holding agreement results across multiple categories or fields.

Return object of interrater.compute_many(). Maps category names to individual AgreementResult instances and provides collection-level utilities.

Parameters:

Name Type Description Default
results dict[str, AgreementResult]

Map of field/category strings to AgreementResult objects.

 required

Attributes:

Name Type Description
fields list[str]

List of field or category names stored in this result.
metrics list[str]

Sorted list of unique metric names evaluated across all fields.

Examples:

>>> multi_res = compute_many(ds, fields=["field_a", "field_b"], metrics=["fleiss"])
>>> multi_res["field_a"].scores
{'fleiss': 0.68}
>>> df = multi_res.to_dataframe()
Source code in interrater/base/result.py 
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
@dataclass(slots=True, frozen=True)
class MultiAgreementResult:
    """
    Immutable container holding agreement results across multiple categories or fields.

    Return object of `interrater.compute_many()`. Maps category names to individual 
    `AgreementResult` instances and provides collection-level utilities.

    Parameters
    ----------
    results : dict[str, AgreementResult]
        Map of field/category strings to `AgreementResult` objects.

    Attributes
    ----------
    fields : list[str]
        List of field or category names stored in this result.
    metrics : list[str]
        Sorted list of unique metric names evaluated across all fields.

    Examples
    --------
    >>> multi_res = compute_many(ds, fields=["field_a", "field_b"], metrics=["fleiss"])
    >>> multi_res["field_a"].scores
    {'fleiss': 0.68}
    >>> df = multi_res.to_dataframe()
    """
    results: dict[str, AgreementResult]

    @property
    def fields(self):
        """Return list of evaluated field/category names."""
        return list(self.results.keys())

    @property
    def metrics(self):
        """Return sorted list of unique metric names evaluated across all fields."""
        metrics = set()

        for result in self.results.values():
            metrics.update(result.metrics)

        return sorted(metrics)

    def to_dataframe(self):
        """
        Convert all field results into a single consolidated pandas DataFrame.

        Returns
        -------
        pd.DataFrame
            Concatenated DataFrame where each row contains its associated `field`.s
        """
        frames = []

        for field, result in self.results.items():
            df = result.to_dataframe()
            df.insert(0, "field", field)
            frames.append(df)

        return pd.concat(frames, ignore_index=True)

    @classmethod
    def concat(cls, *results: "MultiAgreementResult") -> "MultiAgreementResult":
        """
        Combine multiple `MultiAgreementResult` objects into a single result container.

        Parameters
        ----------
        *results : MultiAgreementResult
            One or more `MultiAgreementResult` instances to concatenate.

        Returns
        -------
        MultiAgreementResult
            A unified `MultiAgreementResult` containing all fields.

        Raises
        ------
        ValueError
            If duplicate field names exist across the provided results.

        Examples
        --------
        >>> nominal = compute_many(ds, fields=["field_a", "field_b"], metrics=["fleiss"])
        >>> set = compute_many(ds, fields=["field_a", "field_b"], metrics=["jaccard"])
        >>> combined = MultiAgreementResult.concat(nominal, set)
        """
        combined = {}

        for r in results:
            overlap = combined.keys() & r.results.keys()
            if overlap:
                raise ValueError(f"Duplicate fields found: {sorted(overlap)}")
            combined.update(r.results)

        return cls(combined)

    def extend(self, other: "MultiAgreementResult") -> "MultiAgreementResult":
        """
        Return a new `MultiAgreementResult` combining fields from `self` and `other`.

        Parameters
        ----------
        other : MultiAgreementResult
            Secondary result set to combine with.

        Returns
        -------
        MultiAgreementResult
            A new merged instance.

        Raises
        ------
        ValueError
            If duplicate field names exist between the two objects.

        Examples
        --------
        >>> nominal = compute_many(ds, fields=["field_a", "field_b"], metrics=["fleiss"])
        >>> set = compute_many(ds, fields=["field_a", "field_b"], metrics=["jaccard"])
        >>> combined = nominal.extend(set)
        """
        overlap = self.results.keys() & other.results.keys()

        if overlap:
            raise ValueError(f"Duplicate fields found: {sorted(overlap)}")

        return MultiAgreementResult({
            **self.results,
            **other.results
        })

    def select_metrics(self, metrics: list[str]) -> "MultiAgreementResult":
        """
        Return a new `MultiAgreementResult` filtered to include only selected metrics.

        Parameters
        ----------
        metrics : list[str]
            List of metric names to retain.

        Returns
        -------
        MultiAgreementResult
            Filtered `MultiAgreementResult` instance.
        """
        selected = {}

        for field, result in self.results.items():
            available = [m for m in metrics if m in result.metrics]

            if not available:
                continue

            selected[field] = result.select_metrics(available)

        return MultiAgreementResult(selected)

    def select_fields(self, fields: list[str]) -> "MultiAgreementResult":
        """
        Return a new `MultiAgreementResult` filtered to include only selected fields.

        Parameters
        ----------
        fields : list[str]
            List of field or category names to keep.

        Returns
        -------
        MultiAgreementResult
            Filtered `MultiAgreementResult` instance.

        Raises
        ------
        ValueError
            If none of the requested fields are present.
        """
        selected = {f: self.results[f] for f in fields if f in self.results}

        if not selected:
            raise ValueError(
                f"None of the requested fields are available. "
                f"Available fields: {self.fields}"
            )

        return MultiAgreementResult(selected)

    def aggregate_runs(self) -> pd.DataFrame:
        """
        Aggregate estimates across runs for each model.

        Calculates mean std, min, and max across runs for each metric and model.

        Returns
        -------
        pd.DataFrame
            Summary dataframe grouped by model and metric
        """
        df = self.to_dataframe()

        # Get base model name
        if "field" in df.columns:
            df["model_name"] = df["field"].apply(lambda x: x.split("__run_")[0])
        else:
            df["model_name"] = df["models"]

        # Aggregate numeric metrics across runs
        agg_df = (
            df.groupby(["model_name", "metric"])["score"]
            .agg(
                mean_estimate="mean",
                std_estimate="std",
                min_estimate="min",
                max_estimate="max",
                n_runs="count"
            )
            .reset_index()
        )

        return agg_df.round(4)

    def __iter__(self):
        return iter(self.results.items())

    def __getitem__(self, field):
        return self.results[field]

fields property

Return list of evaluated field/category names.

metrics property

Return sorted list of unique metric names evaluated across all fields.

to_dataframe()

Convert all field results into a single consolidated pandas DataFrame.

Returns:

Type Description
DataFrame

Concatenated DataFrame where each row contains its associated field.s
Source code in interrater/base/result.py 
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
def to_dataframe(self):
    """
    Convert all field results into a single consolidated pandas DataFrame.

    Returns
    -------
    pd.DataFrame
        Concatenated DataFrame where each row contains its associated `field`.s
    """
    frames = []

    for field, result in self.results.items():
        df = result.to_dataframe()
        df.insert(0, "field", field)
        frames.append(df)

    return pd.concat(frames, ignore_index=True)

concat(*results) classmethod

Combine multiple MultiAgreementResult objects into a single result container.

Parameters:

Name Type Description Default
*results MultiAgreementResult

One or more MultiAgreementResult instances to concatenate.

 ()

Returns:

Type Description
MultiAgreementResult

A unified MultiAgreementResult containing all fields.

Raises:

Type Description
ValueError

If duplicate field names exist across the provided results.

Examples:

>>> nominal = compute_many(ds, fields=["field_a", "field_b"], metrics=["fleiss"])
>>> set = compute_many(ds, fields=["field_a", "field_b"], metrics=["jaccard"])
>>> combined = MultiAgreementResult.concat(nominal, set)
Source code in interrater/base/result.py 
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
@classmethod
def concat(cls, *results: "MultiAgreementResult") -> "MultiAgreementResult":
    """
    Combine multiple `MultiAgreementResult` objects into a single result container.

    Parameters
    ----------
    *results : MultiAgreementResult
        One or more `MultiAgreementResult` instances to concatenate.

    Returns
    -------
    MultiAgreementResult
        A unified `MultiAgreementResult` containing all fields.

    Raises
    ------
    ValueError
        If duplicate field names exist across the provided results.

    Examples
    --------
    >>> nominal = compute_many(ds, fields=["field_a", "field_b"], metrics=["fleiss"])
    >>> set = compute_many(ds, fields=["field_a", "field_b"], metrics=["jaccard"])
    >>> combined = MultiAgreementResult.concat(nominal, set)
    """
    combined = {}

    for r in results:
        overlap = combined.keys() & r.results.keys()
        if overlap:
            raise ValueError(f"Duplicate fields found: {sorted(overlap)}")
        combined.update(r.results)

    return cls(combined)

extend(other)

Return a new MultiAgreementResult combining fields from self and other.

Parameters:

Name Type Description Default
other MultiAgreementResult

Secondary result set to combine with.

 required

Returns:

Type Description
MultiAgreementResult

A new merged instance.

Raises:

Type Description
ValueError

If duplicate field names exist between the two objects.

Examples:

>>> nominal = compute_many(ds, fields=["field_a", "field_b"], metrics=["fleiss"])
>>> set = compute_many(ds, fields=["field_a", "field_b"], metrics=["jaccard"])
>>> combined = nominal.extend(set)
Source code in interrater/base/result.py 
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
def extend(self, other: "MultiAgreementResult") -> "MultiAgreementResult":
    """
    Return a new `MultiAgreementResult` combining fields from `self` and `other`.

    Parameters
    ----------
    other : MultiAgreementResult
        Secondary result set to combine with.

    Returns
    -------
    MultiAgreementResult
        A new merged instance.

    Raises
    ------
    ValueError
        If duplicate field names exist between the two objects.

    Examples
    --------
    >>> nominal = compute_many(ds, fields=["field_a", "field_b"], metrics=["fleiss"])
    >>> set = compute_many(ds, fields=["field_a", "field_b"], metrics=["jaccard"])
    >>> combined = nominal.extend(set)
    """
    overlap = self.results.keys() & other.results.keys()

    if overlap:
        raise ValueError(f"Duplicate fields found: {sorted(overlap)}")

    return MultiAgreementResult({
        **self.results,
        **other.results
    })

select_metrics(metrics)

Return a new MultiAgreementResult filtered to include only selected metrics.

Parameters:

Name Type Description Default
metrics list[str]

List of metric names to retain.

 required

Returns:

Type Description
MultiAgreementResult

Filtered MultiAgreementResult instance.
Source code in interrater/base/result.py 
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
def select_metrics(self, metrics: list[str]) -> "MultiAgreementResult":
    """
    Return a new `MultiAgreementResult` filtered to include only selected metrics.

    Parameters
    ----------
    metrics : list[str]
        List of metric names to retain.

    Returns
    -------
    MultiAgreementResult
        Filtered `MultiAgreementResult` instance.
    """
    selected = {}

    for field, result in self.results.items():
        available = [m for m in metrics if m in result.metrics]

        if not available:
            continue

        selected[field] = result.select_metrics(available)

    return MultiAgreementResult(selected)

select_fields(fields)

Return a new MultiAgreementResult filtered to include only selected fields.

Parameters:

Name Type Description Default
fields list[str]

List of field or category names to keep.

 required

Returns:

Type Description
MultiAgreementResult

Filtered MultiAgreementResult instance.

Raises:

Type Description
ValueError

If none of the requested fields are present.
Source code in interrater/base/result.py 
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
def select_fields(self, fields: list[str]) -> "MultiAgreementResult":
    """
    Return a new `MultiAgreementResult` filtered to include only selected fields.

    Parameters
    ----------
    fields : list[str]
        List of field or category names to keep.

    Returns
    -------
    MultiAgreementResult
        Filtered `MultiAgreementResult` instance.

    Raises
    ------
    ValueError
        If none of the requested fields are present.
    """
    selected = {f: self.results[f] for f in fields if f in self.results}

    if not selected:
        raise ValueError(
            f"None of the requested fields are available. "
            f"Available fields: {self.fields}"
        )

    return MultiAgreementResult(selected)

aggregate_runs()

Aggregate estimates across runs for each model.

Calculates mean std, min, and max across runs for each metric and model.

Returns:

Type Description
DataFrame

Summary dataframe grouped by model and metric
Source code in interrater/base/result.py 
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
def aggregate_runs(self) -> pd.DataFrame:
    """
    Aggregate estimates across runs for each model.

    Calculates mean std, min, and max across runs for each metric and model.

    Returns
    -------
    pd.DataFrame
        Summary dataframe grouped by model and metric
    """
    df = self.to_dataframe()

    # Get base model name
    if "field" in df.columns:
        df["model_name"] = df["field"].apply(lambda x: x.split("__run_")[0])
    else:
        df["model_name"] = df["models"]

    # Aggregate numeric metrics across runs
    agg_df = (
        df.groupby(["model_name", "metric"])["score"]
        .agg(
            mean_estimate="mean",
            std_estimate="std",
            min_estimate="min",
            max_estimate="max",
            n_runs="count"
        )
        .reset_index()
    )

    return agg_df.round(4)

EvaluationResult dataclass

Significance evaluation result for a single agreement metric comparison.

Represents hypothesis test outputs generated by interrater.evaluate() for one metric.

Parameters:

Name Type Description Default
metric str

Name of the evaluated agreement metric.

 required
estimate float

Observed estimate (point estimate for one-sample, or difference for two-sample).

 required
p_value float

Permutation test \(p\)-value.

 required
null_distribution ndarray

1D array holding permutation null statistics.

 required
alternative str

Direction of alternative hypothesis ("two-sided", "greater", "less").

 required
method str

Permutation computation method ("exact" or "monte_carlo").

 required
label_a str

Model identifier for result_a.

 required
label_b str or None

Model identifier for result_b if performing a two-sample comparison.

 None

Attributes:

Name Type Description
raters str

Formatted string listing participating models, e.g., "gpt-4, claude-3-5-sonnet".
Source code in interrater/base/result.py 
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
@dataclass(slots=True, frozen=True)
class EvaluationResult:
    """
    Significance evaluation result for a single agreement metric comparison.

    Represents hypothesis test outputs generated by `interrater.evaluate()` for one metric.

    Parameters
    ----------
    metric : str
        Name of the evaluated agreement metric.
    estimate : float
        Observed estimate (point estimate for one-sample, or difference for two-sample).
    p_value : float
        Permutation test $p$-value.
    null_distribution : np.ndarray
        1D array holding permutation null statistics.
    alternative : str
        Direction of alternative hypothesis (`"two-sided"`, `"greater"`, `"less"`).
    method : str
        Permutation computation method (`"exact"` or `"monte_carlo"`).
    label_a : str
        Model identifier for `result_a`.
    label_b : str or None, default=None
        Model identifier for `result_b` if performing a two-sample comparison.

    Attributes
    ----------
    raters : str
        Formatted string listing participating models, e.g., `"gpt-4, claude-3-5-sonnet"`.
    """
    metric: str
    estimate: float
    p_value: float
    null_distribution: np.ndarray

    alternative: str
    method: str

    label_a: str
    label_b: str | None = None

    @property
    def raters(self) -> str:
        """Formatted display string with rater names."""
        return f"{self.label_a}, {self.label_b}"

    def is_significant(self, alpha: float = 0.05) -> bool:
        """
        Check if the hypothesis test is statistically significant at significance level $\\alpha$.

        Parameters
        ----------
        alpha : float, default=0.05
            Significance threshold $\\alpha$.

        Returns
        -------
        bool
            `True` if `p_value < alpha`, `False` otherwise.
        """
        return self.p_value < alpha

raters property

Formatted display string with rater names.

is_significant(alpha=0.05)

Check if the hypothesis test is statistically significant at significance level \(\alpha\).

Parameters:

Name Type Description Default
alpha float

Significance threshold \(\alpha\).

 0.05

Returns:

Type Description
bool

True if p_value < alpha, False otherwise.
Source code in interrater/base/result.py 
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
def is_significant(self, alpha: float = 0.05) -> bool:
    """
    Check if the hypothesis test is statistically significant at significance level $\\alpha$.

    Parameters
    ----------
    alpha : float, default=0.05
        Significance threshold $\\alpha$.

    Returns
    -------
    bool
        `True` if `p_value < alpha`, `False` otherwise.
    """
    return self.p_value < alpha

EvaluationResults dataclass

Container for multiple metric evaluation results from hypothesis testing.

Return object of interrater.evaluate(). Contains a list of EvaluationResult objects and includes convenience methods for filtering, tabular formatting, null distribution plotting and inspection.

Parameters:

Name Type Description Default
results list[EvaluationResult]

Collection of individual metric evaluation results.

 required

Attributes:

Name Type Description
metrics list[str]

Sorted list of unique metric names contained in the results.

Examples:

>>> eval_results = evaluate(res_a, res_b)
>>> eval_results.significant(alpha=0.01)
<EvaluationResults containing 1 significant result>
>>> df = eval_results.to_dataframe()
Source code in interrater/base/result.py 
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
@dataclass(slots=True, frozen=True)
class EvaluationResults:
    """
    Container for multiple metric evaluation results from hypothesis testing.

    Return object of `interrater.evaluate()`. Contains a list of `EvaluationResult` objects 
    and includes convenience methods for filtering, tabular formatting, null distribution 
    plotting and inspection.

    Parameters
    ----------
    results : list[EvaluationResult]
        Collection of individual metric evaluation results.

    Attributes
    ----------
    metrics : list[str]
        Sorted list of unique metric names contained in the results.

    Examples
    --------
    >>> eval_results = evaluate(res_a, res_b)
    >>> eval_results.significant(alpha=0.01)
    <EvaluationResults containing 1 significant result>
    >>> df = eval_results.to_dataframe()
    """
    results: list[EvaluationResult]

    @property
    def metrics(self):
        """Return a sorted list of unique evaluated metric names."""
        return sorted(set(r.metric for r in self.results))

    def __iter__(self):
        return iter(self.results)

    def __len__(self):
        return len(self.results)

    def to_dataframe(self, alpha: float = 0.05):
        """
        Convert evaluation results into a structured pandas DataFrame.

        Parameters
        ----------
        alpha : float, default=0.05
            Significance threshold $\\alpha$ used to populate the boolean `significant` column.

        Returns
        -------
        pd.DataFrame
            DataFrame listing raters, metric, estimate, $p$-value, significance status,
            null mean/std, hypothesis alternative, and computation method.
        """
        return pd.DataFrame([
            {
                "raters": r.raters,
                "metric": r.metric,
                "estimate": r.estimate,
                "p-value": r.p_value,
                "significant": r.p_value < alpha,
                "null_mean": float(np.mean(r.null_distribution)),
                "null_std": float(np.std(r.null_distribution)),
                "alternative": r.alternative,
                "method": r.method
            }
            for r in self.results
        ])

    def select_metrics(self, metrics: list[str]) -> "EvaluationResults":
        """
        Return a new `EvaluationResults` restricted to specified metric names.

        Parameters
        ----------
        metrics : list[str]
            List of metric names to retain.

        Returns
        -------
        EvaluationResults
            New filtered `EvaluationResults` instance.

        Raises
        ------
        ValueError
            If none of the requested metrics exist.
        """
        selected = [r for r in self.results if r.metric in metrics]

        if not selected:
            raise ValueError(
                f"No matching metrics found. "
                f"Available: {self.metrics}"
            )

        return EvaluationResults(selected)

    def significant(self, alpha: float = 0.05) -> "EvaluationResults":
        """
        Filter and return results that meet statistical significance ($p < \\alpha$).

        Parameters
        ----------
        alpha : float, default=0.05
            Significance threshold $\\alpha$.

        Returns
        -------
        EvaluationResults
            A new `EvaluationResults` containing only statistically significant results.
        """
        return EvaluationResults([
            r for r in self.results if r.p_value < alpha
        ])

    def plot_distribution(
        self,
        metric: str,
        bins: int = 30,
        figsize=(7, 5)
    ):
        """
        Plot a histogram of the permutation null distribution alongside the observed estimate.

        Parameters
        ----------
        metric : str
            Name of the metric comparison to plot.
        bins : int, default=30
            Number of histogram bins.
        figsize : tuple[int, int], default=(7, 5)
            Dimensions of the generated figure (width, height).

        Returns
        -------
        plt.Axes
            Matplotlib Axes instance containing the plotted distribution.

        Raises
        ------
        ValueError
            If no comparison is found for the requested metric, or if multiple comparisons
            share the same metric name.
        """
        matches = [r for r in self.results if r.metric == metric]

        if len(matches) == 0:
            raise ValueError(f"No comparison found for metric {metric}")

        if len(matches) > 1:
            raise ValueError("Multiple comparisons found. Filter by rater first")

        result = matches[0]

        fig, ax = plt.subplots(figsize=figsize)

        ax.hist(result.null_distribution, bins=bins)

        ax.axvline(
            result.estimate,
            linestyle="--",
            label="Observed"
        )
        ax.set_title(f"{result.metric}: {result.raters}")
        ax.set_xlabel("Permutation test statistic")
        ax.set_ylabel("Frequency")
        ax.legend()
        return ax

    def inspect(self, metric: str, tolerance: float = 1e-10) -> pd.DataFrame:
        """
        Inspect properties of the null distribution for a specific metric.

        Parameters
        ----------
        metric : str
            Name of the metric to inspect.
        tolerance : float, default=1e-10
            Absolute tolerance used when counting null values close to the observed statistic.

        Returns
        -------
        pd.DataFrame
            DataFrame summarizing observed statistic, null min/max/mean/std, and count of 
            null permutations near observed value.

        Raises
        ------
        ValueError
            If the metric does not uniquely identify exactly one comparison result.
        """
        results = [r for r in self.results if r.metric == metric]

        if len(results) != 1:
            raise ValueError("Metric must identify exactly one comparison")

        r = results[0]
        null = r.null_distribution

        return pd.DataFrame({
            "value": [
                "observed",
                "null_min",
                "null_max",
                "null_mean",
                "null_std",
                "near_observed_count"
            ],
            "statistic": [
                r.estimate,
                np.min(null),
                np.max(null),
                np.mean(null),
                np.std(null),
                np.sum(np.isclose(null, r.estimate, atol=tolerance))
            ]
        })

metrics property

Return a sorted list of unique evaluated metric names.

to_dataframe(alpha=0.05)

Convert evaluation results into a structured pandas DataFrame.

Parameters:

Name Type Description Default
alpha float

Significance threshold \(\alpha\) used to populate the boolean significant column.

 0.05

Returns:

Type Description
DataFrame

DataFrame listing raters, metric, estimate, \(p\)-value, significance status, null mean/std, hypothesis alternative, and computation method.
Source code in interrater/base/result.py 
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
def to_dataframe(self, alpha: float = 0.05):
    """
    Convert evaluation results into a structured pandas DataFrame.

    Parameters
    ----------
    alpha : float, default=0.05
        Significance threshold $\\alpha$ used to populate the boolean `significant` column.

    Returns
    -------
    pd.DataFrame
        DataFrame listing raters, metric, estimate, $p$-value, significance status,
        null mean/std, hypothesis alternative, and computation method.
    """
    return pd.DataFrame([
        {
            "raters": r.raters,
            "metric": r.metric,
            "estimate": r.estimate,
            "p-value": r.p_value,
            "significant": r.p_value < alpha,
            "null_mean": float(np.mean(r.null_distribution)),
            "null_std": float(np.std(r.null_distribution)),
            "alternative": r.alternative,
            "method": r.method
        }
        for r in self.results
    ])

select_metrics(metrics)

Return a new EvaluationResults restricted to specified metric names.

Parameters:

Name Type Description Default
metrics list[str]

List of metric names to retain.

 required

Returns:

Type Description
EvaluationResults

New filtered EvaluationResults instance.

Raises:

Type Description
ValueError

If none of the requested metrics exist.
Source code in interrater/base/result.py 
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
def select_metrics(self, metrics: list[str]) -> "EvaluationResults":
    """
    Return a new `EvaluationResults` restricted to specified metric names.

    Parameters
    ----------
    metrics : list[str]
        List of metric names to retain.

    Returns
    -------
    EvaluationResults
        New filtered `EvaluationResults` instance.

    Raises
    ------
    ValueError
        If none of the requested metrics exist.
    """
    selected = [r for r in self.results if r.metric in metrics]

    if not selected:
        raise ValueError(
            f"No matching metrics found. "
            f"Available: {self.metrics}"
        )

    return EvaluationResults(selected)

significant(alpha=0.05)

Filter and return results that meet statistical significance (\(p < \alpha\)).

Parameters:

Name Type Description Default
alpha float

Significance threshold \(\alpha\).

 0.05

Returns:

Type Description
EvaluationResults

A new EvaluationResults containing only statistically significant results.
Source code in interrater/base/result.py 
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
def significant(self, alpha: float = 0.05) -> "EvaluationResults":
    """
    Filter and return results that meet statistical significance ($p < \\alpha$).

    Parameters
    ----------
    alpha : float, default=0.05
        Significance threshold $\\alpha$.

    Returns
    -------
    EvaluationResults
        A new `EvaluationResults` containing only statistically significant results.
    """
    return EvaluationResults([
        r for r in self.results if r.p_value < alpha
    ])

plot_distribution(metric, bins=30, figsize=(7, 5))

Plot a histogram of the permutation null distribution alongside the observed estimate.

Parameters:

Name Type Description Default
metric str

Name of the metric comparison to plot.

 required
bins int

Number of histogram bins.

 30
figsize tuple[int, int]

Dimensions of the generated figure (width, height).

 (7, 5)

Returns:

Type Description
Axes

Matplotlib Axes instance containing the plotted distribution.

Raises:

Type Description
ValueError

If no comparison is found for the requested metric, or if multiple comparisons share the same metric name.
Source code in interrater/base/result.py 
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
def plot_distribution(
    self,
    metric: str,
    bins: int = 30,
    figsize=(7, 5)
):
    """
    Plot a histogram of the permutation null distribution alongside the observed estimate.

    Parameters
    ----------
    metric : str
        Name of the metric comparison to plot.
    bins : int, default=30
        Number of histogram bins.
    figsize : tuple[int, int], default=(7, 5)
        Dimensions of the generated figure (width, height).

    Returns
    -------
    plt.Axes
        Matplotlib Axes instance containing the plotted distribution.

    Raises
    ------
    ValueError
        If no comparison is found for the requested metric, or if multiple comparisons
        share the same metric name.
    """
    matches = [r for r in self.results if r.metric == metric]

    if len(matches) == 0:
        raise ValueError(f"No comparison found for metric {metric}")

    if len(matches) > 1:
        raise ValueError("Multiple comparisons found. Filter by rater first")

    result = matches[0]

    fig, ax = plt.subplots(figsize=figsize)

    ax.hist(result.null_distribution, bins=bins)

    ax.axvline(
        result.estimate,
        linestyle="--",
        label="Observed"
    )
    ax.set_title(f"{result.metric}: {result.raters}")
    ax.set_xlabel("Permutation test statistic")
    ax.set_ylabel("Frequency")
    ax.legend()
    return ax

inspect(metric, tolerance=1e-10)

Inspect properties of the null distribution for a specific metric.

Parameters:

Name Type Description Default
metric str

Name of the metric to inspect.

 required
tolerance float

Absolute tolerance used when counting null values close to the observed statistic.

 1e-10

Returns:

Type Description
DataFrame

DataFrame summarizing observed statistic, null min/max/mean/std, and count of null permutations near observed value.

Raises:

Type Description
ValueError

If the metric does not uniquely identify exactly one comparison result.
Source code in interrater/base/result.py 
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
def inspect(self, metric: str, tolerance: float = 1e-10) -> pd.DataFrame:
    """
    Inspect properties of the null distribution for a specific metric.

    Parameters
    ----------
    metric : str
        Name of the metric to inspect.
    tolerance : float, default=1e-10
        Absolute tolerance used when counting null values close to the observed statistic.

    Returns
    -------
    pd.DataFrame
        DataFrame summarizing observed statistic, null min/max/mean/std, and count of 
        null permutations near observed value.

    Raises
    ------
    ValueError
        If the metric does not uniquely identify exactly one comparison result.
    """
    results = [r for r in self.results if r.metric == metric]

    if len(results) != 1:
        raise ValueError("Metric must identify exactly one comparison")

    r = results[0]
    null = r.null_distribution

    return pd.DataFrame({
        "value": [
            "observed",
            "null_min",
            "null_max",
            "null_mean",
            "null_std",
            "near_observed_count"
        ],
        "statistic": [
            r.estimate,
            np.min(null),
            np.max(null),
            np.mean(null),
            np.std(null),
            np.sum(np.isclose(null, r.estimate, atol=tolerance))
        ]
    })

BootstrapResult dataclass

Immutable container for standard error and confidence interval bounds derived from bootstrapping.

Parameters:

Name Type Description Default
standard_error float

Estimated standard error of the agreement statistic.

 required
ci_lower float

Lower bound of the non-parametric BCa confidence interval.

 required
ci_upper float

Upper bound of the non-parametric BCa confidence interval.

 required
Source code in interrater/base/result.py 
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
@dataclass(slots=True, frozen=True)
class BootstrapResult:
    """
    Immutable container for standard error and confidence interval bounds derived from bootstrapping.

    Parameters
    ----------
    standard_error : float
        Estimated standard error of the agreement statistic.
    ci_lower : float
        Lower bound of the non-parametric BCa confidence interval.
    ci_upper : float
        Upper bound of the non-parametric BCa confidence interval.
    """
    standard_error: float
    ci_lower: float
    ci_upper: float

PermutationResult dataclass

Output summary of a raw permutation test run.

Parameters:

Name Type Description Default
observed_difference float

The observed test statistic or score difference.

 required
p_value float

Calculated \(p\)-value under the permutation null distribution.

 required
null_distribution ndarray

Array containing all null test statistics computed during permutation resampling.

 required
alternative str

Direction of hypothesis test ("two-sided", "greater", or "less").

 required
method str

Resampling execution method ("exact" or "monte_carlo").

 required
n_resamples int

Number of permutations evaluated.

 required
Source code in interrater/base/result.py 
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
@dataclass(slots=True, frozen=True)
class PermutationResult:
    """
    Output summary of a raw permutation test run.

    Parameters
    ----------
    observed_difference : float
        The observed test statistic or score difference.
    p_value : float
        Calculated $p$-value under the permutation null distribution.
    null_distribution : np.ndarray
        Array containing all null test statistics computed during permutation resampling.
    alternative : str
        Direction of hypothesis test (`"two-sided"`, `"greater"`, or `"less"`).
    method : str
        Resampling execution method (`"exact"` or `"monte_carlo"`).
    n_resamples : int
        Number of permutations evaluated.
    """
    observed_difference: float
    p_value: float
    null_distribution: np.ndarray
    alternative: str
    method: str
    n_resamples: int

:::