intercoder_reliability: Intercoder reliability for LLM annotations

In localLLM: Running Local LLMs with 'llama.cpp' Backend

Intercoder reliability for LLM annotations

Description

Intercoder reliability for LLM annotations

Usage

intercoder_reliability(
  annotations,
  method = c("auto", "cohen", "krippendorff"),
  label_levels = NULL,
  sample_col = "sample_id",
  model_col = "model_id",
  label_col = "label"
)

Arguments

annotations	Output from [explore()] or a compatible data frame with at least 'sample_id', 'model_id', and 'label' columns.
method	One of '"auto"', '"cohen"', or '"krippendorff"'. The '"auto"' setting computes both pairwise Cohen's Kappa and Krippendorff's Alpha (when applicable).
label_levels	Optional factor levels to enforce a consistent ordering in the resulting tables.
sample_col	Column name that identifies samples when 'annotations' is a user-provided data frame.
model_col	Column name for the model identifier when using a custom 'annotations' data frame.
label_col	Column name containing model predictions when using a custom 'annotations' data frame.

Value

A list containing 'cohen' (data frame of pairwise kappas) and/or 'krippendorff' (overall alpha statistic with per-item agreement scores).

localLLM documentation built on Nov. 3, 2026, 1:09 a.m.