Statistics

Estimating uncertainty¶

Uncertainty is estimated in the package by computing the standard error and bootstrapped confidence intervals of the point estimate, if specified as a parameter when calling compute(), compute_many(), or compare_to_ground_truth().

Confidence intervals are calculated using bias-corrected and accelerated (BCa) bootstrapping, a nonparametric bootstrap method. The bias-correction parameter adjusts for over- or under-estimation introduced by resampling, and is related to the proportion of bootstrap estimates that are less than the observed statistic. The acceleration parameter is proportional to the skewness (asymmetry) of the bootstrap distribution, and is generally estimated using the jackknife (leave one out) method. We use BCa to ensure the robustness of estimated confidence intervals, given that rating data may be skewed.

Resampling is done via cluster bootstrapping, as rated items may be nested or dependent on other fields. Particularly with language model structured outputs, items to resample are often complex object types. For example, in the model output below, we would want to resample the whole set of "recurring_daily_patterns", not merely each 'time window, pattern, rank' object or each individual field, e.g., time window.

"recurring_daily_patterns": [
    {
      "time_window": "Afternoon (12pm - 5pm)",
      "pattern": "Hyperglycemia",
      "rank": 1
    },
    {
      "time_window": "Evening (5pm - 9pm)",
      "pattern": "Hyperglycemia",
      "rank": 2
    },
    {
      "time_window": "Late evening (9pm - 12am)",
      "pattern": "Hyperglycemia",
      "rank": 3
    }
],

Note that bootstrapping confidence intervals assumes the sample size is large enough to be representative of the population, at minimum typically \(n \geq 30\) items. If the sample size is not sufficiently large, bootstrap estimates are likely to be biased and have less precision. Smaller sample sizes may benefit from parametric bootstrap methods, which are not currently implemented in the package.

Evaluating significance¶

Statistical significance is estimated in the package using permutation tests for specified hypotheses when calling evaluate().

Hypotheses¶

Both two-sample and one-sample hypotheses are supported. The direction of the alternative hypothesis \(H_a\) is specified in the alternative parameter with options being "two-sided" (default), "greater", or "less".

A two-sample comparison with alternative="two-sided", for example, tests the following hypotheses,

\[ \begin{align} H_0 &: \theta_A - \theta_B = 0 \\ H_a &: \theta_A - \theta_B \neq 0 \end{align} \]

where \(\theta\) is any agreement coefficient evaluated on raters \(A\) and \(B\). These hypotheses could be used to test, e.g., whether the difference between models' internal agreement is statistically significant, or whether the agreement between one model and the ground truth is statistically different than the agreement between another model and the ground truth.

A one-sample comparison with alternative="greater" tests the following hypotheses,

\[ \begin{align} H_0 &: \theta_A = 0 \\ H_a &: \theta_A > 0 \end{align} \]

where \(\theta\) is any agreement coefficient evaluated on one rater \(A\). These hypotheses could be used to test e.g., whether a model's internal consistency is statistically significant across multiple runs, if the extent of agreement between a model and the ground truth is statistically significant, or if the extent of agreement between models is statistically significant.

Permutation methods¶

There are two methods in the package that could be used to run permutation tests. An exact permutation test, with no random sampling, calculates the statistic across every possible combination of the data (\(n!\) for \(n\) items) which quickly becomes computationally infeasible as the sample size increases. Monte Carlo sampling is thus often used for feasibility, which uses random sampling without replacement to evaluate only a subset of possible permutations.

Note that with Monte Carlo sampling, the \(p\)-value for a permutation test is approximate, while no random sampling yields a \(p\)-value that is exact.

The permutation method can be specified in a call to evaluate() with the method parameter, where options are "exact", "monte_carlo", or "auto" (default). The "auto" option does Monte Carlo sampling by default but switches to the exact method if the total possible permutations of the data are less than the specified max_exact_permutations parameter, which would depend on one's computational resources.

Null model resampling depends on the number of result objects passed to evaluate(): a two-sample comparison uses a swap-based null model, where item rows are swapped between the datasets of two result objects, and a one-sample comparison uses a shuffle-based null model, where labels within items are shuffled.

References¶

Bootstrapping (statistics), Wikipedia, https://en.wikipedia.org/wiki/Bootstrapping_%28statistics%29.
Wicklin, R. (2017). The bias-corrected and accelerated (BCa) bootstrap interval. https://blogs.sas.com/content/iml/2017/07/12/bootstrap-bca-interval.html.
Permutation test, Wikipedia, https://en.wikipedia.org/wiki/Permutation_test.
Wilber, J. (2019). The permutation test. https://www.jwilber.me/permutationtest/.