murmuration: Links case, hospital, outbreak, event linelists, vaccination...

In starling: Record Linkage for Public Health Surveillance

Links case, hospital, outbreak, event linelists, vaccination datasets

Description

**Bird note**: A murmuration is one of nature's most arresting phenomena: tens of thousands of starlings moving as a single fluid shape across the sky, with no conductor and no central instruction — each bird responding only to its nearest neighbours until a coherent, shifting pattern emerges from the whole flock. murmuration() works the same way: it takes two separate, uncoordinated datasets and lets pairwise local comparisons between records resolve, through the Fellegi-Sunter EM algorithm, into one coherent linked result. No single record "knows" about the others — the shape emerges from the scoring.

Probabilistic record linkage of two surveillance datasets using the Fellegi-Sunter framework. Links diagnostic case linelists, hospital admission records, outbreak linelists, and event manifests to vaccination history (Australian Immunisation Register) or to each other. Wraps the full reclin2 pipeline — blocking, comparison, EM scoring, threshold selection, and post-linkage window filtering — in a single practitioner- facing call.

Usage

murmuration(
  df1,
  df2,
  linkage_type = NULL,
  event_type = NULL,
  method = "probabilistic",
  event_date = NULL,
  id_var,
  blocking_var,
  compare_vars,
  threshold_value = 17,
  perch_before_linking = FALSE,
  vax_window = list(days_before = 14, days_after = 0, lookback_days = Inf),
  cohort_window = NULL,
  days_allowed_before_event = 7,
  days_allowed_after_event = 14,
  one_row_per_person = TRUE,
  clean_eggs = TRUE,
  days_between_onset_death = 30,
  last_follow_up = NULL
)

Arguments

df1	This is a dataframe object, cleaned using clean_the_nest, and would often represent the base, or "x" dataset (when doing left joins). Typically this would be a dataset of cases, have enough data to create linkages, and have event dates (e.g., onset_date).
df2	This is a dataframe object, cleaned using clean_the_nest, and would often represent the admissions or vaccination dataset ("y" dataset when doing left joins). Typically this would have enough data to create linkages, and include either admission data or vaccination event data (e.g. Australian Immunization Register).
linkage_type	The linkage to perform. As of starling 1.1.0 the preferred value is "cohort2event" — a single, explicit linkage concept: link a base linelist of people (a "cohort" — cases, hospitalisations, a birth cohort, an outbreak linelist, a flight manifest) to a stream of dated events (a "case" diagnosis, a "hospitalisation", or a "vaccination"), keeping events that fall in a time window relative to an anchor. Pair it with event_type to name the event stream. The window may sit before, during, or after the cohort's follow-up, governed by vax_window (event-relative) or cohort_window (calendar). Legacy codes (deprecated, still functional): the historical "c2h"/"v2c"/"v2h"/"v2e" tokens still work and route identically, but emit a deprecation warning nudging to the collapsed form: "c2h" (cases->hospitalisations) becomes event_type = "hospitalisation"; "v2c"/"v2h" (->vaccination history) become event_type = "vaccination"; "v2e" (fixed-date event) becomes event_type = "vaccination" with a scalar Date passed to event_date. If NULL (default), "c2h" is assumed for backwards compatibility. When linking to a vaccination dataset, use a single row per person (run the vaccination data through clean_the_nest() with lie_nest_flat = TRUE).
event_type	Required when linkage_type = "cohort2event": names the event stream in df2. One of "vaccination", "hospitalisation" ("hospitalization" also accepted), or "case". Ignored (with the routing inferred from the code) when a legacy linkage_type is supplied.
method	Linkage engine. "probabilistic" (default) uses the Fellegi-Sunter EM pipeline (reclin2) with fuzzy field comparison and a score threshold — appropriate when identifiers are imperfect and you need weighted, partial-agreement matching. "deterministic" skips EM scoring entirely and links only records whose compare_vars match exactly (all fields, no partial credit). Deterministic linkage is the right choice when you have a clean, reliable composite key (e.g. full name + date of birth, as in Smoll et al. 2023) and want reproducible, explainable, threshold-free links. Under "deterministic": weights is NA (no composite score exists), threshold_value/perch_before_linking are ignored, and a non-unique key in df2 emits a message rather than silently resolving. All linkage types and both window mechanisms work under either method.
event_date	The anchor date used to define the linkage window and determine valid vaccinations or related admissions. For "c2h", "v2c", and "v2h": a character string naming any date column present in df1. Any date column produced by mudnester::clean_the_nest() is valid here — not just "onset_date" or "admission_date". In birth-cohort or coverage studies, pass "cohort_entry_date" to anchor the vaccination window from the start of follow-up, or "cohort_exit_date" to bound from the end. Examples: "onset_date", "admission_date", "cohort_entry_date", "cohort_exit_date". For "v2e": a Date object (e.g., as.Date("2024-12-15")) representing a fixed event date shared across all records (e.g. a flight, an outbreak event). This parameter is critical for determining valid vaccinations (must occur before event_date - days_allowed_before_event) and for "c2h" linkages, determining disease-related admissions.
id_var	Variable name (e.g. "id"). This is critical for data-linkage and the base dataset is the dataset you would left join onto (e.g. the "x" dataset). Cannot have missing data, or the observation will be lost in the linking process.
blocking_var	Variable name (e.g. "block2"). Choice of blocking variable. Use flock() to automatically construct up to three blocking variables (coarse, medium, fine) from standardised linkage fields — it produces columns named block1, block2, block3 by default. You can also create your own. Choose a broader/looser variable (e.g. gender) for small datasets or low data quality, and a finer variable (e.g. postcode + DOB year) for large, high-quality datasets.
compare_vars	Vector of variables. Used to compare variables between each dataset and calculate the string score differences. Typically names, dates of births and medicare/social security numbers.
threshold_value	Numeric (e.g. 17), default is 17. This is the Fellegi-Sunter log-likelihood ratio score above which a pair is classified as a match. The score is dataset-specific — there is no universal correct value. Use murmuration_plot() to inspect your score distribution and select a threshold that sits in the natural valley between the match and non-match peaks. Higher values increase specificity (fewer false links); lower values increase sensitivity (fewer missed true matches). Typical working ranges against SCPHU datasets with 4-5 comparison variables: conservative ~20-25; balanced ~15-20; sensitive ~10-14.
perch_before_linking	Logical. If TRUE, calls perch on the scored pairs immediately after the EM algorithm fits and before the threshold is applied, printing a sensitivity table showing match counts and clerical burden across the score range. In interactive sessions the plot is also displayed. In non-interactive sessions (e.g. Quarto render, batch jobs) only the table is printed and execution continues automatically. Default FALSE.
vax_window	A named list defining the vaccination validity window relative to event_date. Used for all vaccination linkage types ("v2c", "v2h", "v2e"). Contains up to three elements: days_before Minimum days before event_date a vaccine must have been administered to count as prior valid exposure. Vaccines administered within this buffer (i.e. fewer than days_before days before the event) are excluded because they may not yet have conferred protection. Default 14. Set to 0 if you want all prior vaccinations regardless of recency (e.g. birth-cohort studies where protection is assumed immediate). days_after Maximum days after event_date to retain a vaccination record. Default 0 (no post-event vaccinations retained). Set to a positive value to capture post-event vaccination – useful for studying vaccination uptake following a diagnosis or intervention (e.g. 365 to capture all vaccinations in the year after diagnosis). Post-event vaccinations are flagged with a new vax_timing column ("pre_event" or "post_event") so they can be analysed separately downstream. lookback_days Maximum days before event_date to look back for valid vaccinations. Default Inf (no upper lookback limit – any prior vaccination regardless of age counts). Set to 365 to restrict to vaccinations in the past year only (appropriate for annual influenza VE studies where a vaccine from five years ago is not the relevant exposure). Example: vax_window = list(days_before = 14, days_after = 0, lookback_days = 365) retains only vaccinations that occurred 14 or more days before the event and within the past year, excluding both too-recent and too-old doses. Overridden by cohort_window when both are supplied. For "c2h" linkages, use days_allowed_before_event and days_allowed_after_event instead (those govern the admission window, not vaccination).
cohort_window	An optional named list defining a calendar-based observation window that replaces vax_window when supplied. Use this for cohort studies, coverage analyses, or any design where the observation window is defined at the study level rather than relative to individual event dates: entry A Date object (or character coercible to Date via as.Date()) giving the start of the observation window. Vaccinations before this date are excluded. Example: as.Date("2024-01-01"). exit A Date object giving the end of the observation window. Vaccinations after this date are excluded. Example: as.Date("2024-12-31"). When cohort_window is supplied, vax_window is ignored entirely and a message is emitted confirming which window is active. Useful when the event date is unknown, unreliable, or irrelevant to the vaccination exposure definition (e.g. a prevalence survey, a screening programme evaluation, or a birth-cohort study anchored to a fixed follow-up period rather than individual birth dates). Can be used with any linkage_type.
days_allowed_before_event	Numeric. For "c2h" linkages only: the lower bound of the disease-related admission window – how many days before event_date an admission may still be considered disease-related. Default 7. Not used for vaccination linkage types; use vax_window$days_before for those.
days_allowed_after_event	Numeric. For "c2h" linkages only: the upper bound of the disease-related admission window – how many days after event_date an admission may still be considered disease-related. Default 14. Not used for vaccination linkage types; use vax_window$days_after for those.
one_row_per_person	Logical (TRUE or FALSE) with the default being TRUE. It will take multiple admissions per person, and create a series of variables prefixed with "first_", such as "first_admission_date", and put into a single row all admission events, and create a series of variables suffixed with "s", such as "admission_dates". Will work with single admissions per person.
clean_eggs	Logical (TRUE or FALSE) with the default being TRUE. Drops all the .y variables that are duplicates of the second dataset (df2), and keeps the variables and removes the .x from df1. If you leave this on, many, if not most variables will have ".x" or ".y" attached to them (e.g. gender) and thus keep this as TRUE for default, and FALSE if you want to check the linkages are true and working.
days_between_onset_death	Numeric (e.g. "30"). If you have put a date of death into the clean_the_nest command (which will rename it to dod), then the command will find disease related dates of death. This is chosen number of days between event_date and death for a disease-related death. Often this may be 30 days for SARS-CoV-2 or can be much longer for HIV. If you don't want an upper limit, use "9999".
last_follow_up	Represents a date (input as ymd(2024-11-22)) that represents last follow-up. This could be the latest admission date of a dataset. Used for calculating survival time.

Details

Make sure that you do not have the same variables (other than linkage variables e.g. letternames, DOB, gender) in both datasets. Always make sure your date columns are properly formatted using as.Date(). For example, if both datasets have a date of death, choose the dataset with the highest confidence and drop the date of death from the other before linking. If linking to a hospitalisation dataset, the difference between event_date and admission_date is used to identify disease-related admissions; unrelated hospitalisations can be filtered separately using ICD-10 codes or AR-DRG codes prior to linkage.

The recommended pre-linkage workflow is:

1. mudnester::clean_the_nest() to clean and prep data for linkage. Pay close attention to your linkage variables (letternames, date of birth, medicare number, gender and/or postcode), and ensure all dates are formatted as dates.

2. linkage_quality() to run a structured battery of pre-linkage quality checks across both datasets (completeness, duplicates, Medicare validity, date plausibility, gender and DOB year distribution).

3. check_medicare to validate Medicare check digits and flag invalid numbers before they enter comparison scoring.

4. flock() to create up to three blocking variables at different specificity levels (coarse, medium, fine) — pass one to the blocking_var argument below.

5. murmuration with linkage_type="v2c" to link cases to vaccination data.

6. murmuration with linkage_type="v2h" to link a v2c dataset to hospitalization data. Or skip linking to case data, and just build a v2h dataset for test-negative case-control studies.

7. murmuration with linkage_type="v2e" to link event linelists (flight manifests, outbreak investigations) to vaccination history.

8. murmuration_plot() to visualise the full linkage score distribution and confirm the threshold before accepting the linked dataset.

9. mudnester::preening() to prettify the dataframe prepping it for exploration, analysis and presentation. Great to use with gtsummary::tbl_summary().

## On threshold selection The threshold_value is a Fellegi-Sunter log-likelihood ratio score and is dataset-specific — there is no universal correct value. The default of 17 was chosen empirically against SCPHU datasets with 4-5 comparison variables (names, DOB, Medicare, gender). Use flock_plot() to inspect the bimodal score distribution from your specific dataset and identify the natural valley between the match and non-match peaks. Typical working ranges: conservative (high specificity) ~20-25; balanced (default) ~15-20; sensitive (high recall) ~10-14.

Value

A linked dataset with some new variables.

Note

Ensure there are no missing vaccination dates in vaccination dataset prior to murmuration. Murmuration requires complete vaccination data (equal date and type columns per observation) to achieve correct matching of vaccination columns. If there are too few variables to match on, then matching will not work well. For example, if you have first name, last name and date of birth, and a very large dataset (Immunization Register), then the scoring will not differentiate true from false matches. Consider deterministic linkage when there is a paucity of information to use to derive linkage scores.

Examples

## Not run: 
# Example 1: Link cases to vaccination history (onset_date as anchor)
dx_clean <- clean_the_nest(dx_data,
  data_type = "cases",
  id_var = "identity",
  lettername1 = "first_name",
  lettername2 = "surname",
  dob = "date_of_birth",
  gender = "gender",
  postcode = "postcode",
  medicare = "medicare_no",
  diagnosis = "disease_name",
  onset_date = "date_of_onset")

vax_clean <- clean_the_nest(vax_data,
  data_type = "vaccination",
  id_var = "patient_id",
  lettername1 = "firstname",
  lettername2 = "last_name",
  dob = "birth_date",
  gender = "gender",
  postcode = "postcode",
  medicare = "medicare_number",
  vax_type = "vaccine_delivered",
  vax_date = "service_date")

df1 <- murmuration(dx_clean, vax_clean,
  linkage_type = "v2c",
  event_date = "onset_date",
  id_var = "identity",
  blocking_var = "gender",
  compare_vars = c("lettername1", "lettername2", "dob"),
  clean_eggs = FALSE)

# Example 2: Link hospitalization data to vaccination history (admission_date as anchor)
hosp_clean <- clean_the_nest(hosp_data,
  data_type = "hospital",
  id_var = "patient_id",
  lettername1 = "firstname",
  lettername2 = "last_name",
  dob = "birth_date",
  gender = "sex",
  postcode = "zip_codes",
  medicare = "medicare_number",
  admission_date = "date_of_admission",
  discharge_date = "date_of_discharge")

df2 <- murmuration(hosp_clean, vax_clean,
  linkage_type = "v2h",
  event_date = "admission_date",
  id_var = "patient_id",
  blocking_var = "gender",
  compare_vars = c("lettername1", "lettername2", "medicare10", "dob"),
  clean_eggs = FALSE,
  one_row_per_person = TRUE)

# Example 3: Link cases to hospitalisations (onset_date as anchor)
df3 <- murmuration(dx_clean, hosp_clean,
  linkage_type = "c2h",
  event_date = "onset_date",
  id_var = "identity",
  blocking_var = "postcode",
  compare_vars = c("lettername1", "lettername2", "dob", "medicare10"),
  days_allowed_before_event = 7,
  days_allowed_after_event = 30,
  clean_eggs = FALSE)

# Example 4: Birth-cohort study (e.g. RICOR nirsevimab) — cohort_window approach.
# Captures any vaccination administered during the defined follow-up period,
# regardless of individual event dates.
cohort_clean <- clean_the_nest(birth_cohort_data,
  data_type = "cases",
  id_var = "baby_id",
  lettername1 = "first_name",
  lettername2 = "last_name",
  dob = "babys_date_of_birth",
  gender = "sex")

df_cohort <- murmuration(cohort_clean, vax_clean,
  linkage_type  = "v2c",
  event_date    = "dob",
  id_var        = "baby_id",
  blocking_var  = "gender",
  compare_vars  = c("lettername1", "lettername2", "dob"),
  cohort_window = list(
    entry = as.Date("2023-01-01"),
    exit  = as.Date("2024-06-30")
  ),
  clean_eggs = FALSE)

# Example 4b: Annual influenza VE study — 1-year lookback, 14-day buffer.
# Only vaccinations in the 12 months before onset, and >= 14 days before onset,
# are retained as valid prior exposure.
df_flu_ve <- murmuration(cases_clean, vax_clean,
  linkage_type = "v2c",
  event_date   = "onset_date",
  id_var       = "identity",
  blocking_var = "gender",
  compare_vars = c("lettername1", "lettername2", "dob", "medicare10"),
  vax_window   = list(
    days_before   = 14,
    days_after    = 0,
    lookback_days = 365
  ),
  clean_eggs = FALSE)

# Example 4c: Post-diagnosis vaccination uptake study.
# Retains vaccinations up to 1 year AFTER diagnosis as well as valid prior doses.
# The vax_timing column flags each dose as "pre_event" or "post_event".
df_post_dx <- murmuration(cases_clean, vax_clean,
  linkage_type = "v2c",
  event_date   = "onset_date",
  id_var       = "identity",
  blocking_var = "gender",
  compare_vars = c("lettername1", "lettername2", "dob", "medicare10"),
  vax_window   = list(
    days_before   = 14,
    days_after    = 365,
    lookback_days = Inf
  ),
  clean_eggs = FALSE)

# Example 5: Link flight manifest to vaccination history (fixed event Date object)
manifest_clean <- clean_the_nest(manifest_data,
  data_type = "cases",
  id_var = "passenger_id",
  lettername1 = "first_name",
  lettername2 = "surname",
  dob = "date_of_birth",
  gender = "gender")

df_flight <- murmuration(manifest_clean, vax_clean,
  linkage_type = "v2e",
  event_date = as.Date("2024-03-15"),
  id_var = "passenger_id",
  blocking_var = "gender",
  compare_vars = c("lettername1", "lettername2", "dob"),
  days_allowed_before_event = 14,
  clean_eggs = FALSE)

# Example 6: Link outbreak linelist to vaccination history (fixed event Date object)
linelist_clean <- clean_the_nest(linelist_data,
  data_type = "cases",
  id_var = "case_id",
  lettername1 = "first_name",
  lettername2 = "surname",
  dob = "date_of_birth",
  gender = "gender",
  postcode = "postcode",
  medicare = "medicare_no",
  onset_date = "onset_date")

df_outbreak <- murmuration(linelist_clean, vax_clean,
  linkage_type = "v2e",
  event_date = as.Date("2024-06-01"),
  id_var = "case_id",
  blocking_var = "postcode",
  compare_vars = c("lettername1", "lettername2", "dob", "medicare10"),
  days_allowed_before_event = 7,
  clean_eggs = FALSE)

## End(Not run)

starling documentation built on July 10, 2026, 9:07 a.m.