llm_judge: LLM-as-a-Judge Evaluation

In LLMR: Interface for Large Language Model APIs in R

LLM-as-a-Judge Evaluation

Description

Evaluates outputs in a target column against a custom prompt using llm_mutate_tags() for clean tag-based extraction. The target column value is available in the prompt template as {.target}.

Usage

llm_judge(
  .data,
  .target,
  .config,
  prompt,
  .tags = c("reasoning", "score"),
  .output = "judge_res",
  ...
)

Arguments

.data	Data frame of experiment results.
.target	Bare column name containing the output to evaluate.
.config	The judge llm_config.
prompt	Evaluation prompt template. Use {.target} to reference the target column value (other data columns are also available).
.tags	Tags to extract from the judge response. Defaults to c("reasoning", "score").
.output	Name of the column that receives the judge's raw response. Default "judge_res".
...	Passed to llm_mutate_tags().

Value

.data with judge output columns appended.

See Also

llm_mutate_tags(), llm_parse_tags()

Examples

## Not run: 
results |>
  llm_judge(
    .target = response_text,
    .config = judge_cfg,
    prompt = "Rate this answer on a 1-5 scale:\n{.target}",
    .tags = c("reasoning", "score")
  )

## End(Not run)

LLMR documentation built on July 11, 2026, 1:06 a.m.