dot-filter_near_duplicates.aidR: Remove near-duplicate entries from different formats

In aidR: Reading and Standardizing Automated Insulin Delivery Data

Remove near-duplicate entries from different formats

Description

The same entry exported by two platforms may differ slightly, e.g. in the timestamp (rounding), in the value (rounding) or because one platform provides more details (e.g. a label). Two entries of the same data type are near-duplicates if they

• come from different formats

• have the same unit

• lie at most max_minutes apart

• differ in value by at most the tolerance of the data type (carbs: 1 g, SMBG: 1 mg/dL, bolus: 0.05 U, basal: 0.05 U/h).

Basal rates change frequently, so that neighbouring entries often have similar rates. For basal, two entries must therefore lie at most max_minutes_basal apart and their duration must differ by at most tolerance_basal_duration. Entries with a missing duration are never considered near-duplicates. Of each pair of duplicates, one entry is kept: the one with a link_id, otherwise the one with more information. Its timestamp and value are used, and its missing information (e.g. a label) is filled from the other entry. If information was filled, both formats are joined with a / (e.g. "glooko/tidepool").

Usage

.filter_near_duplicates.aidR(
  data_id,
  max_minutes = 5,
  tolerance_carbs = 1,
  tolerance_SMBG = 1,
  tolerance_bolus = 0.05,
  tolerance_basal = 0.05,
  tolerance_basal_duration = 1/60,
  max_minutes_basal = 2
)

Arguments

data_id	A list with all data found for one individual (cleaned), with one entry per data type.
max_minutes	Maximum time difference (in minutes) between two duplicates.
tolerance_carbs	Tolerance for detecting duplicate carb entries, default 1 g.
tolerance_SMBG	Tolerance for detecting duplicate SMBG entries, default 1 mg/dL.
tolerance_bolus	Tolerance for detecting duplicate bolus entries, default 0.05 U.
tolerance_basal	Tolerance for detecting duplicate basal rates, default 0.05 U/h.
tolerance_basal_duration	Tolerance for the duration of duplicate basal entries, in hours, default 1 minute.
max_minutes_basal	Maximum time difference (in minutes) between two duplicate basal entries.

Details

A warning is given if the entries of two formats match better when shifted by one hour, see .check_time_shift.aidR.

Value

data_id without the duplicates. Removed rows are logged in the attribute removed, see removed_entries.

aidR documentation built on Oct. 28, 2026, 5:13 p.m.