foundry_agreement: Compute agreement metrics for LLM annotation
In foundryR: Data Frame Workflows for 'Microsoft Foundry'

foundry_agreement

R Documentation

Compute agreement metrics for LLM annotation

Description

Compare model labels with human or gold-standard labels using common publication-friendly metrics: accuracy, macro precision/recall/F1, Cohen's kappa, and Krippendorff's alpha (using irr when installed, otherwise a base-R nominal implementation).

Usage

foundry_agreement(data, estimate, truth)

Arguments

`data`	Data frame containing estimates and truth.
`estimate`	Character. Column name with model labels.
`truth`	Character. Column name with reference labels.

Value

A tibble with one row per metric.

Examples

labels <- data.frame(
  model = c("yes", "no", "yes"),
  human = c("yes", "no", "no")
)
foundry_agreement(labels, estimate = "model", truth = "human")

foundryR documentation built on Sept. 25, 2026, 1:10 a.m.