Skip to content

interrater-lm

interrater-lm is a Python package for evaluating language model-style structured outputs with agreement coefficients, including statistical procedures for deriving uncertainty and significance.

The aim of this package is not primarily to provide novel implementations of agreement coefficients but instead to make computing these simpler for comparing language models. To that end, we make use of existing libraries for some definitions, and define our own only when well-maintained implementations do not exist or significant additional preprocessing is needed to use with language model-style structured outputs.