Data

Dataset class to standardize input data, with tools for cleaning, processing, viewing metadata, and comparing to ground truth

`Dataset` `dataclass` ¶

Standardized container for input data.

Enforces required structural columns (item, model, run) and target annotations. Provides utilities for column mapping, slicing, level of measurement handling, and ground-truth integration.

Parameters:

Name	Type	Description	Default
`_df`	`DataFrame`	Long dataframe with model output data	required
`target`	`str`	Name of the column containing the annotations to compare	required
`level`	`str \| LevelType`	Level of measurement of the target responses	`NOMINAL`
`mapping`	`dict[str, str]`	Column mapping overrides for non-standard schema names, e.g., `{"item": "question_id", "model": "llm_name"}`	`dict()`

Attributes:

Name	Type	Description
`target`	`str`	Active target annotation column name
`level`	`LevelType`	Parsed level of measurement
`mapping`	`dict[str, str]`	Active schema column mappings

Examples:

>>> import pandas as pd
>>> from interrater import Dataset
>>> df = pd.DataFrame({
...     "item": [1, 1, 2, 2],
...     "model": ["gpt-4", "claude", "gpt-4", "claude"],
...     "run": [0, 0, 0, 0],
...     "rating": ["A", "A", "B", "A"]
... })
>>> dataset = Dataset(df, target="rating", level="nominal")
>>> dataset
Dataset(n=4, target='rating', level='nominal', models=2, items=2)

Source code in interrater/base/dataset.py

@dataclass(slots=True)
class Dataset:
    """
    Standardized container for input data.

    Enforces required structural columns (`item`, `model`, `run`) and target
    annotations. Provides utilities for column mapping, slicing, level of 
    measurement handling, and ground-truth integration.

    Parameters
    ---------
    _df: pd.DataFrame
        Long dataframe with model output data
    target: str
        Name of the column containing the annotations to compare
    level: str or LevelType, default=LevelType.NOMINAL
        Level of measurement of the target responses
    mapping: dict[str, str], default = {}
        Column mapping overrides for non-standard schema names,
        e.g., `{"item": "question_id", "model": "llm_name"}`

    Attributes
    ----------
    target: str
        Active target annotation column name
    level: LevelType
        Parsed level of measurement
    mapping: dict[str, str]
        Active schema column mappings

    Examples
    --------
    >>> import pandas as pd
    >>> from interrater import Dataset
    >>> df = pd.DataFrame({
    ...     "item": [1, 1, 2, 2],
    ...     "model": ["gpt-4", "claude", "gpt-4", "claude"],
    ...     "run": [0, 0, 0, 0],
    ...     "rating": ["A", "A", "B", "A"]
    ... })
    >>> dataset = Dataset(df, target="rating", level="nominal")
    >>> dataset
    Dataset(n=4, target='rating', level='nominal', models=2, items=2)
    """
    _df: pd.DataFrame
    target: str
    level: str | LevelType = LevelType.NOMINAL # default is nominal
    mapping: dict[str, str] = field(default_factory=dict)

    def __post_init__(self) -> None:
        self.level = LevelType(self.level)

        # Map dataset columns to user data columns
        default_map = {"item": "item", "model": "model", "run": "run"}
        self.mapping = {**default_map, **self.mapping}

        self._validate()

    @classmethod
    def _load_df(cls, source: PathOrDataFrame) -> pd.DataFrame:
        """
        Parse raw data into a pandas DataFrame

        Parameters
        ---------
        source: PathOrDataFrame
            File path (`.csv`, `.json`), existing DataFrame, dictionary, or list

        Returns
        -------
        pd.DataFrame
            Constructed or copy of DataFrame from source

        Raises
        -----
        ValueError
            If given an unsupported file extension
        TypeError
            If given an incompatible object type
        """
        if isinstance(source, pd.DataFrame):
            return source.copy()
        elif isinstance(source, str):
            if source.endswith(".csv"):
                return pd.read_csv(source)
            elif source.endswith(".json"):
                return pd.read_json(source)
            else:
                raise ValueError("Unsupported file format. Provide a .csv or .json path.")
        elif isinstance(source, (dict, list)):
            return pd.DataFrame(source)
        else:
            raise TypeError("Source must be a file path, DataFrame, dict, or list")

    @classmethod
    def from_source(cls, source: PathOrDataFrame, target: str, level: str | LevelType = LevelType.NOMINAL) -> Dataset:
        """
        Construct a `Dataset` directly from a file path, list, dictionary, or DataFrame

        Parameters
        ----------
        source : PathOrDataFrame
            Input source as a `.csv`/`.json` path, dictionary, list, or DataFrame.
        target : str
            Name of the column containing annotations.
        level : str or LevelType, default=LevelType.NOMINAL
            Measurement scale level (`"nominal"`, `"ordinal"`, `"interval"`, or `"ratio"`).

        Returns
        -------
        Dataset
            An initialized `Dataset` instance.
        """
        df = cls._load_df(source)
        return cls(df, target=target, level=level)

    def with_ground_truth(self, source: PathOrDataFrame, target: str) -> Dataset:
        """
        Injects external ground-truth dataset as a reserved rater

        Matches ground-truth annotations to items in the current data by `item` ID,
        appends the matching rows, and returns a new `Dataset` instance

        Parameters
        ---------
        source: PathOrDataFrame
            Ground-truth data source as a `.csv`/`.json` path, dictionary, list, or DataFrame.
        target: str
            Name of the column containing annotations.

        Returns
        -------
        Dataset
            A new `Dataset` instance containing both model and ground-truth annotations

        Raises
        ------
        ValueError
            - If `source` lacks an `'item'` column or the specified `target` column.
            - If zero items match between model outputs and ground truth.
        """
        gt_df = self._load_df(source)

        if "item" not in gt_df.columns:
            raise ValueError("The ground truth dataset must contain an 'item' column to map ratings")
        if target not in gt_df.columns:
            raise ValueError(f"Target column {target} not found in ground truth dataset")

        # Get strict mapping
        gt_map = gt_df.set_index("item")[target].to_dict()

        # Build rows for ground truth across all unique items
        gt_rows = []
        for item in self.items:
            if item in gt_map:
                gt_rows.append({
                    "item": item,
                    "model": "ground_truth",
                    "run": 0,
                    self.target: gt_map[item]
                })

        if not gt_rows:
            raise ValueError("Zero matching items found between the model outputs and the ground truth dataset.")

        combined_df = pd.concat([self._df, pd.DataFrame(gt_rows)], ignore_index=True)

        return Dataset(combined_df, target=self.target, level=self.level)

    @classmethod
    def from_csv(cls, path: str, target: str) -> "Dataset":
        """
        Convenience constructor to load a dataset directly from a CSV file.

        Parameters
        ----------
        path : str
            Path to the target CSV file.
        target : str
            Column name containing ratings or outputs.

        Returns
        -------
        Dataset
            An initialized `Dataset` instance.
        """
        return cls(pd.read_csv(path), target=target)

    @property
    def raters(self):
        """
        Return unique rater pairs defined by `(model, run)` tuples.

        Returns
        -------
        list[tuple[str, int]]
            Unique combinations of models and run identifiers.
        """
        return list(self.df[["model", "run"]].drop_duplicates().itertuples(index=False, name=None))

    @property
    def df(self) -> pd.DataFrame:
        """
        Return the standardized underlying DataFrame with mapped schema names.

        Returns
        -------
        pd.DataFrame
            Renamed DataFrame enforcing canonical column names (`item`, `model`, `run`, target).
        """
        _map = {v: k for k, v in self.mapping.items()}
        _map[self.target] = self.target

        return self._df[list(_map.keys())].rename(columns=_map)

    def copy(self) -> "Dataset":
        """
        Return a deep copy of the Dataset and its underlying DataFrame.

        Returns
        -------
        Dataset
            Cloned `Dataset` instance.
        """
        return Dataset(self._df.copy(), target=self.target, level=self.level)

    def slice(self, **filters: Any) -> "Dataset":
        """
        Filter rows by arbitrary metadata or column values.

        Supports single values, lists, tuples, or sets for filtering by category.

        Parameters
        ----------
        **filters : Any
            Column name and target value(s) to slice on.

        Returns
        -------
        Dataset
            A new `Dataset` instance containing the filtered subset of rows.

        Raises
        ------
        KeyError
            If any specified filter column does not exist in the DataFrame.

        Examples
        --------
        >>> subset = dataset.slice(category="clinical_vignettes")
        >>> subset = dataset.slice(model=["gpt-4o", "claude-3-5-sonnet"])
        """
        df = self._df

        for column, value in filters.items():
            if column not in df.columns:
                raise KeyError(f"Unknown column: '{column}'")

            if isinstance(value, (list, tuple, set)):
                df = df[df[column].isin(value)]
            else:
                df = df[df[column] == value]

        return Dataset(df.reset_index(drop=True), target=self.target, level=self.level)

    def unique(self, column: str) -> list:
         """
        Return sorted unique non-null values from a specified column.

        Parameters
        ----------
        column : str
            Target column name in the DataFrame.

        Returns
        -------
        list
            Sorted list of unique non-null values.

        Raises
        ------
        KeyError
            If the requested column does not exist.
         """
         if column not in self._df.columns:
              raise KeyError(f"Unknown column: '{column}'")

         return sorted(self._df[column].dropna().unique().tolist())

    @property
    def models(self) -> list[str]:
        """
        Return a list of unique models present in the dataset.

        Examples
        --------
        >>> from interrater import Dataset
        >>> dataset = Dataset(df, target="rating", level="nominal")
        >>> dataset.models
        ['gpt-5.4-mini', 'gpt-5.4-nano']
        """
        return self.unique("model")

    @property
    def runs(self) -> list[int]:
        """
        Return a list of unique run identifiers present in the dataset.

        Examples
        --------
        >>> from interrater import Dataset
        >>> dataset = Dataset(df, target="rating", level="nominal")
        >>> dataset.runs
        [0, 1]
        """
        return self.unique("run")

    @property
    def items(self):
        """
        Return a list of unique item identifiers present in the dataset.

        Examples
        --------
        >>> from interrater import Dataset
        >>> dataset = Dataset(df, target="rating", level="nominal")
        >>> dataset.items
        <StringArray>
        ['108052_daily_pattern_1',
         '108052_daily_pattern_2',
         '108052_daily_pattern_3',
         '108052_findings_1',
         '108052_findings_2',
         '108052_findings_3',
         '108052_findings_4',
         '108052_findings_5',
         '108052_top_recs']
        Length: 23, dtype: str
        """
        return self._df["item"].unique()

    @property
    def metadata_columns(self) -> list[str]:
        """
        List non-required metadata column names present in the dataset.

        Examples
        --------
        >>> from interrater import Dataset
        >>> dataset = Dataset(df, target="rating", level="nominal")
        >>> dataset.metadata_columns
        [category]
        """
        return [c for c in self._df.columns if c not in REQUIRED_COLS | {self.target}]

    @property
    def is_nominal(self) -> bool:
        """True if measurement scale level is nominal."""
        return self.level == LevelType.NOMINAL

    @property
    def is_ordinal(self) -> bool:
        """True if measurement scale level is ordinal."""
        return self.level == LevelType.ORDINAL

    @property
    def is_interval(self) -> bool:
        """True if measurement scale level is interval."""
        return self.level == LevelType.INTERVAL

    @property
    def is_ratio(self) -> bool:
        """True if measurement scale level is ratio."""
        return self.level == LevelType.RATIO

    def _validate(self) -> None:
        """
        Validate schema column presence and uniqueness constraints.

        Raises
        ------
        ValueError
            If mapped required columns are missing or if duplicate (item, model, run) 
            entries are present.
        """
        columns = set(self._df.columns)

        # Validate with mapped column names
        mapped_item = self.mapping["item"]
        mapped_model = self.mapping["model"]
        mapped_run = self.mapping["run"]

        missing = {mapped_item, mapped_model, mapped_run, self.target} - columns
        if missing:
            raise ValueError(f"Required columns missing from dataframe: {sorted(missing)}")

        duplicates = self.df.duplicated(subset=[mapped_item, mapped_model, mapped_run])
        if duplicates.any():
            raise ValueError("Duplicate (item, model, run) entries detected")

    def __len__(self) -> int:
        return len(self._df)

    def __repr__(self) -> str:
        return (
            f"Dataset("
            f"n={len(self)}, "
            f"target='{self.target}', "
            f"level='{self.level}', "
            f"models={len(self.models)}, "
            f"items={len(self.items)})"
        )

`from_source(source, target, level=LevelType.NOMINAL)` `classmethod` ¶

Construct a Dataset directly from a file path, list, dictionary, or DataFrame

Parameters:

Name	Type	Description	Default
`source`	`PathOrDataFrame`	Input source as a `.csv`/`.json` path, dictionary, list, or DataFrame.	required
`target`	`str`	Name of the column containing annotations.	required
`level`	`str or LevelType`	Measurement scale level (`"nominal"`, `"ordinal"`, `"interval"`, or `"ratio"`).	`LevelType.NOMINAL`

Returns:

Type	Description
`Dataset`	An initialized `Dataset` instance.

Source code in interrater/base/dataset.py

@classmethod
def from_source(cls, source: PathOrDataFrame, target: str, level: str | LevelType = LevelType.NOMINAL) -> Dataset:
    """
    Construct a `Dataset` directly from a file path, list, dictionary, or DataFrame

    Parameters
    ----------
    source : PathOrDataFrame
        Input source as a `.csv`/`.json` path, dictionary, list, or DataFrame.
    target : str
        Name of the column containing annotations.
    level : str or LevelType, default=LevelType.NOMINAL
        Measurement scale level (`"nominal"`, `"ordinal"`, `"interval"`, or `"ratio"`).

    Returns
    -------
    Dataset
        An initialized `Dataset` instance.
    """
    df = cls._load_df(source)
    return cls(df, target=target, level=level)

`with_ground_truth(source, target)` ¶

Injects external ground-truth dataset as a reserved rater

Matches ground-truth annotations to items in the current data by item ID, appends the matching rows, and returns a new Dataset instance

Parameters:

Name	Type	Description	Default
`source`	`PathOrDataFrame`	Ground-truth data source as a `.csv`/`.json` path, dictionary, list, or DataFrame.	required
`target`	`str`	Name of the column containing annotations.	required

Returns:

Type	Description
`Dataset`	A new `Dataset` instance containing both model and ground-truth annotations

Raises:

Type	Description
`ValueError`	If `source` lacks an `'item'` column or the specified `target` column. If zero items match between model outputs and ground truth.

Source code in interrater/base/dataset.py

def with_ground_truth(self, source: PathOrDataFrame, target: str) -> Dataset:
    """
    Injects external ground-truth dataset as a reserved rater

    Matches ground-truth annotations to items in the current data by `item` ID,
    appends the matching rows, and returns a new `Dataset` instance

    Parameters
    ---------
    source: PathOrDataFrame
        Ground-truth data source as a `.csv`/`.json` path, dictionary, list, or DataFrame.
    target: str
        Name of the column containing annotations.

    Returns
    -------
    Dataset
        A new `Dataset` instance containing both model and ground-truth annotations

    Raises
    ------
    ValueError
        - If `source` lacks an `'item'` column or the specified `target` column.
        - If zero items match between model outputs and ground truth.
    """
    gt_df = self._load_df(source)

    if "item" not in gt_df.columns:
        raise ValueError("The ground truth dataset must contain an 'item' column to map ratings")
    if target not in gt_df.columns:
        raise ValueError(f"Target column {target} not found in ground truth dataset")

    # Get strict mapping
    gt_map = gt_df.set_index("item")[target].to_dict()

    # Build rows for ground truth across all unique items
    gt_rows = []
    for item in self.items:
        if item in gt_map:
            gt_rows.append({
                "item": item,
                "model": "ground_truth",
                "run": 0,
                self.target: gt_map[item]
            })

    if not gt_rows:
        raise ValueError("Zero matching items found between the model outputs and the ground truth dataset.")

    combined_df = pd.concat([self._df, pd.DataFrame(gt_rows)], ignore_index=True)

    return Dataset(combined_df, target=self.target, level=self.level)

`from_csv(path, target)` `classmethod` ¶

Convenience constructor to load a dataset directly from a CSV file.

Parameters:

Name	Type	Description	Default
`path`	`str`	Path to the target CSV file.	required
`target`	`str`	Column name containing ratings or outputs.	required

Returns:

Type	Description
`Dataset`	An initialized `Dataset` instance.

Source code in interrater/base/dataset.py

@classmethod
def from_csv(cls, path: str, target: str) -> "Dataset":
    """
    Convenience constructor to load a dataset directly from a CSV file.

    Parameters
    ----------
    path : str
        Path to the target CSV file.
    target : str
        Column name containing ratings or outputs.

    Returns
    -------
    Dataset
        An initialized `Dataset` instance.
    """
    return cls(pd.read_csv(path), target=target)

`raters` `property` ¶

Return unique rater pairs defined by (model, run) tuples.

Returns:

Type	Description
`list[tuple[str, int]]`	Unique combinations of models and run identifiers.

`df` `property` ¶

Return the standardized underlying DataFrame with mapped schema names.

Returns:

Type	Description
`DataFrame`	Renamed DataFrame enforcing canonical column names (`item`, `model`, `run`, target).

`copy()` ¶

Return a deep copy of the Dataset and its underlying DataFrame.

Returns:

Type	Description
`Dataset`	Cloned `Dataset` instance.

Source code in interrater/base/dataset.py

def copy(self) -> "Dataset":
    """
    Return a deep copy of the Dataset and its underlying DataFrame.

    Returns
    -------
    Dataset
        Cloned `Dataset` instance.
    """
    return Dataset(self._df.copy(), target=self.target, level=self.level)

`slice(**filters)` ¶

Filter rows by arbitrary metadata or column values.

Supports single values, lists, tuples, or sets for filtering by category.

Parameters:

Name	Type	Description	Default
`**filters`	`Any`	Column name and target value(s) to slice on.	`{}`

Returns:

Type	Description
`Dataset`	A new `Dataset` instance containing the filtered subset of rows.

Raises:

Type	Description
`KeyError`	If any specified filter column does not exist in the DataFrame.

Examples:

>>> subset = dataset.slice(category="clinical_vignettes")
>>> subset = dataset.slice(model=["gpt-4o", "claude-3-5-sonnet"])

Source code in interrater/base/dataset.py

def slice(self, **filters: Any) -> "Dataset":
    """
    Filter rows by arbitrary metadata or column values.

    Supports single values, lists, tuples, or sets for filtering by category.

    Parameters
    ----------
    **filters : Any
        Column name and target value(s) to slice on.

    Returns
    -------
    Dataset
        A new `Dataset` instance containing the filtered subset of rows.

    Raises
    ------
    KeyError
        If any specified filter column does not exist in the DataFrame.

    Examples
    --------
    >>> subset = dataset.slice(category="clinical_vignettes")
    >>> subset = dataset.slice(model=["gpt-4o", "claude-3-5-sonnet"])
    """
    df = self._df

    for column, value in filters.items():
        if column not in df.columns:
            raise KeyError(f"Unknown column: '{column}'")

        if isinstance(value, (list, tuple, set)):
            df = df[df[column].isin(value)]
        else:
            df = df[df[column] == value]

    return Dataset(df.reset_index(drop=True), target=self.target, level=self.level)

`unique(column)` ¶

Return sorted unique non-null values from a specified column.

Parameters:

Name	Type	Description	Default
`column`	`str`	Target column name in the DataFrame.	required

Returns:

Type	Description
`list`	Sorted list of unique non-null values.

Raises:

Type	Description
`KeyError`	If the requested column does not exist.

Source code in interrater/base/dataset.py

def unique(self, column: str) -> list:
     """
    Return sorted unique non-null values from a specified column.

    Parameters
    ----------
    column : str
        Target column name in the DataFrame.

    Returns
    -------
    list
        Sorted list of unique non-null values.

    Raises
    ------
    KeyError
        If the requested column does not exist.
     """
     if column not in self._df.columns:
          raise KeyError(f"Unknown column: '{column}'")

     return sorted(self._df[column].dropna().unique().tolist())

`models` `property` ¶

Return a list of unique models present in the dataset.

Examples:

>>> from interrater import Dataset
>>> dataset = Dataset(df, target="rating", level="nominal")
>>> dataset.models
['gpt-5.4-mini', 'gpt-5.4-nano']

`runs` `property` ¶

Return a list of unique run identifiers present in the dataset.

Examples:

>>> from interrater import Dataset
>>> dataset = Dataset(df, target="rating", level="nominal")
>>> dataset.runs
[0, 1]

`items` `property` ¶

Return a list of unique item identifiers present in the dataset.

Examples:

>>> from interrater import Dataset
>>> dataset = Dataset(df, target="rating", level="nominal")
>>> dataset.items
<StringArray>
['108052_daily_pattern_1',
 '108052_daily_pattern_2',
 '108052_daily_pattern_3',
 '108052_findings_1',
 '108052_findings_2',
 '108052_findings_3',
 '108052_findings_4',
 '108052_findings_5',
 '108052_top_recs']
Length: 23, dtype: str

`metadata_columns` `property` ¶

List non-required metadata column names present in the dataset.

Examples:

>>> from interrater import Dataset
>>> dataset = Dataset(df, target="rating", level="nominal")
>>> dataset.metadata_columns
[category]

`is_nominal` `property` ¶

True if measurement scale level is nominal.

`is_ordinal` `property` ¶

True if measurement scale level is ordinal.

`is_interval` `property` ¶

True if measurement scale level is interval.

`is_ratio` `property` ¶

True if measurement scale level is ratio.

:::

Data

Dataset dataclass ¶

from_source(source, target, level=LevelType.NOMINAL) classmethod ¶

with_ground_truth(source, target) ¶

from_csv(path, target) classmethod ¶

raters property ¶

df property ¶

copy() ¶

slice(**filters) ¶

unique(column) ¶

models property ¶

runs property ¶

items property ¶

metadata_columns property ¶

is_nominal property ¶

is_ordinal property ¶

is_interval property ¶

is_ratio property ¶

`Dataset` `dataclass` ¶

`from_source(source, target, level=LevelType.NOMINAL)` `classmethod` ¶

`with_ground_truth(source, target)` ¶

`from_csv(path, target)` `classmethod` ¶

`raters` `property` ¶

`df` `property` ¶

`copy()` ¶

`slice(**filters)` ¶

`unique(column)` ¶

`models` `property` ¶

`runs` `property` ¶

`items` `property` ¶

`metadata_columns` `property` ¶

`is_nominal` `property` ¶

`is_ordinal` `property` ¶

`is_interval` `property` ¶

`is_ratio` `property` ¶