dragon_synthesize_pairs: Build preference pairs from a model's own samples

In dragonfarm: Fine-Tune Small Language Models with LoRA from R

Build preference pairs from a model's own samples

Description

Preference optimization needs, for each prompt, a better and a worse reply. This function makes them without hand labelling, in one of two ways:

Usage

dragon_synthesize_pairs(
  prompts,
  student,
  judge = NULL,
  teacher = NULL,
  n_samples = 4,
  min_gap = 2,
  rubric = NULL,
  system = NULL,
  temperature = 0.8,
  max_new_tokens = 256,
  file = NULL,
  runs_dir = dragon_runs_dir(),
  name = "preferences"
)

Arguments

prompts	A character vector of prompts, or a mapped dragon_dataset whose prompt template is rendered for every row. See dragon_prompts() for prompts from an existing run, which is how distillation from "the model's own prompts" is done: pass dragon_prompts(run, "train").
student	The model whose replies are being improved: normally the dragon_run you will continue from. Also a model id, ellmer chat, or function.
judge	Scores the student's samples. See dragon_judge() for what is accepted. Required unless teacher is given.
teacher	Writes the chosen reply instead of judging. See dragon_synthesize().
n_samples	Samples per prompt in judge mode. At least 2.
min_gap	Minimum score difference between chosen and rejected in judge mode. Pairs below it are dropped.
rubric	What the judge should value. See dragon_judge().
system	System prompt for the teacher. Also stored with every row so the student trains with the same instruction.
temperature	Sampling temperature for the student. Needs to be above zero in judge mode, or every sample is the same.
max_new_tokens	Reply length cap for local teachers.
file	Where to write the JSONL. Defaults to a timestamped file under ⁠synth/⁠ in runs_dir.
runs_dir	Parent directory for the default file.
name	Label used in the default file name.

Details

• With a judge: the student answers each prompt n_samples times at a non-zero temperature, the judge scores every sample from 1 to 10, and the best and worst become the chosen and rejected replies. Prompts whose samples are too close (min_gap) or identical are dropped. This is the loop behind RLAIF-style training: the model improves on its own outputs under a judge's preferences.

• With a teacher: the teacher's reply is chosen and the student's is rejected. Cheap and effective when the teacher is clearly stronger.

The result is saved as JSONL and returned as a dataset mapped with dragon_map_pairs(), ready for dragon_prefer(), usually continuing from the student run itself.

Value

A dragon_dataset mapped as preference pairs. In judge mode the file also records chosen_score and rejected_score.

Examples

if (interactive()) {
# Close the loop: sample from the fine-tuned run, let a judge rank, train DPO on the result.
pairs <- dragon_synthesize_pairs(dragon_prompts(sft, "train", n = 200), student = sft,
                                 judge = dragon_judge_anthropic(model = "claude-sonnet-5"))
dpo <- dragon_prefer(pairs, sft, wait = TRUE)
dragon_judge(dpo, against = "base", judge = dragon_judge_anthropic())
}

dragonfarm documentation built on Oct. 12, 2026, 5:07 p.m.