audit_disagreements: Surface strong LLM-human disagreements for a manual audit

In screenllm: LLM-Assisted Title/Abstract Screening for Systematic Reviews

Surface strong LLM-human disagreements for a manual audit

Description

Returns records where the LLM ensemble and the human reviewer disagree at high confidence. On one of the benchmark reviews reported in the paper, a similar audit caught genuine screener errors in 28\ disagreements. This is intended as a low-cost quality-control step after the main screen.

Usage

audit_disagreements(
  ranked,
  decisions,
  strong_fp_score = 70,
  strong_fn_score = 30
)

Arguments

ranked	A ranking (output of rank_records()).
decisions	A tibble with columns id and human_decision.
strong_fp_score	LLM score at or above which an accept is "confident" (default 70).
strong_fn_score	LLM score below which a reject is "confident" (default 30).

Value

A tibble of disagreement rows, one per record.

screenllm documentation built on Sept. 24, 2026, 5:11 p.m.