find_duplicates: Detect and remove duplicate records

In screenllm: LLM-Assisted Title/Abstract Screening for Systematic Reviews

Detect and remove duplicate records

Description

Two records are considered duplicates if they share a non-missing DOI (case-insensitive) or if their normalised titles match. Title normalisation drops punctuation, lowercases, collapses whitespace, and (optionally) fuzzy-matches with stringdist if that package is available. Returns the input tibble with an added duplicate_of column: NA for unique records, otherwise the id of the earlier record that duplicates it.

Usage

find_duplicates(records, fuzzy = TRUE)

Arguments

records	A tibble of records from read_records().
fuzzy	Whether to fuzzy-match titles (Jaro-Winkler similarity of at least 0.95). Requires the stringdist package; falls back to exact normalised-title match if stringdist is not installed.

Details

Multi-database searches (Scopus, Web of Science, Google Scholar) often produce 10-20 percent duplicates; running this on the fresh corpus before ranking avoids scoring the same abstract three or four times.

Value

The input tibble with a new duplicate_of column.

Examples

recs <- data.frame(
  id = c("a", "b", "c"),
  title = c("Coral reefs", "Coral Reefs.", "Deep sea"),
  abstract = c("x", "x", "y")
)
find_duplicates(recs)

screenllm documentation built on Sept. 24, 2026, 5:11 p.m.