make_calibration_data: Compute Calibration Statistics for Eyewitness Identification

View source: R/calibration_functions.R

make_calibration_dataR Documentation

Compute Calibration Statistics for Eyewitness Identification

Description

Computes calibration analysis metrics following Juslin, Olsson, & Winman (1996). Calibration analysis assesses the match between confidence and accuracy, providing a more appropriate measure than simple confidence-accuracy correlation.

Usage

make_calibration_data(
  data,
  confidence_bins = NULL,
  choosers_only = TRUE,
  lineup_size = 6,
  confidence_scale = c("auto", "0-1", "0-100"),
  innocent_suspect = "auto",
  population = NULL,
  tp_fillers = c("exclude", "error")
)

Arguments

data

A dataframe with the following columns:

  • target_present: Logical. TRUE if guilty suspect in lineup

  • identification: Character. "suspect", "filler", or "reject"

  • confidence: Numeric. Confidence rating

confidence_bins

Numeric vector of bin edges for grouping confidence (e.g., c(0, 60, 80, 100) creates bins 0-60, 60-80, 80-100). If NULL, uses individual confidence levels.

choosers_only

Logical. If TRUE (default), only analyze suspect identifications (population = "suspect_ids"). FALSE is deprecated: it now means population = "choosers". Ignored when population is supplied.

lineup_size

Integer. Number of people in lineup (default = 6). Used for estimating incorrect suspect IDs from filler choices in target-absent lineups.

confidence_scale

How the confidence scale is interpreted: "auto" (default; max <= 1 treated as 0-1, otherwise as 0-100, with a warning for ambiguous Likert/0-10 scales), "0-1", or "0-100". C and O/U require a 0-1/0-100 probability scale.

innocent_suspect

How innocent-suspect identifications in target-absent lineups are obtained. "designated": the lineups had a designated innocent suspect, so target-absent "suspect" rows are counted directly, and zero such rows means a false-ID rate of zero. "estimated": there was no designated innocent suspect, so the innocent-suspect count is estimated as filler IDs divided by lineup_size. "auto" (the default) chooses "designated" when any target-absent row is a suspect ID or the data have a non-missing suspect_pos column, and "estimated" otherwise, with a message.

population

Which responses are calibrated: "suspect_ids" (suspect identifications, as in CAC analysis; correct when target-present), "choosers" (witnesses who identify someone; only target-present suspect IDs are correct; see tp_fillers), or "nonchoosers" (rejections; correct when target-absent). If NULL (default), it is set from choosers_only.

tp_fillers

For population = "choosers" only: how filler identifications in target-present lineups are treated. "exclude" (default) leaves them out, as Brewer and Wells (2006) did, because a filler is known in advance not to be the culprit. "error" keeps them as incorrect identifications; Brewer and Wells note that overconfidence is then more marked.

Details

Calibration analysis distinguishes between:

  • Calibration (C): How well confidence matches accuracy (perfect = 0)

  • Over/underconfidence (O/U): Overall bias in confidence judgments (0 = no net bias in the mean; overconfident and underconfident bins can cancel, so perfect calibration needs C = 0)

  • Resolution (NRI): Ability to discriminate correct from incorrect responses

The calibration statistic C measures the weighted mean squared deviation between mean confidence and accuracy in each bin:

C = \sum_{j=1}^{J} \frac{n_j}{N} (c_j - a_j)^2

The Normalized Resolution Index (NRI) captures how well confidence discriminates correct from incorrect responses:

NRI = \frac{\frac{1}{N} \sum_{j=1}^{J} n_j (a_j - \bar{a})^2}{\bar{a}(1-\bar{a})}

The default population, suspect identifications, is the one used in CAC analysis (Mickes, 2015). In the eyewitness calibration literature (e.g., Brewer & Wells, 2006) choosers are witnesses who identify someone. Brewer and Wells excluded filler identifications from target-present lineups, because a filler is known not to be the culprit, and scored all target-absent identifications as errors; population = "choosers" reproduces that analysis, and tp_fillers = "error" keeps the target-present filler identifications as errors instead. Nonchoosers (population = "nonchoosers") are witnesses who reject the lineup; a rejection is correct when the culprit is absent.

If target-absent data contain designated innocent-suspect identifications, those observations are used directly. Otherwise, each target-absent filler identification contributes weight 1 / lineup_size as an estimate of innocent-suspect identifications. In that latter case the per-bin counts and effective total used by C, O/U, NRI, and downstream ANRI can be fractional; the results are lineup-size approximations rather than statistics from an observed designated-suspect outcome.

Value

A list containing:

  • calibration_data: Dataframe with per-bin confidence, accuracy, and counts

  • C: Calibration statistic (weighted mean squared difference between confidence and accuracy)

  • OU: Over/underconfidence (mean confidence minus mean accuracy)

  • NRI: Normalized Resolution Index, the between-bin variance of accuracy divided by \bar{a}(1 - \bar{a}), the total variance of the outcome (the proportion of outcome variance explained by the confidence bins)

  • overall_accuracy: Mean accuracy across all responses

  • overall_confidence: Mean confidence across all responses

  • n_total: Total number of responses analyzed

References

Mickes, L. (2015). Receiver operating characteristic analysis and confidence-accuracy characteristic analysis in investigations of system variables and estimator variables that affect eyewitness memory. Journal of Applied Research in Memory and Cognition, 4(2), 93-102.

Juslin, P., Olsson, N., & Winman, A. (1996). Calibration and diagnosticity of confidence in eyewitness identification: Comments on what can be inferred from the low confidence-accuracy correlation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22(5), 1304-1316.

Brewer, N., & Wells, G. L. (2006). The confidence-accuracy relationship in eyewitness identification: Effects of lineup instructions, foil similarity, and target-absent base rates. Journal of Experimental Psychology: Applied, 12(1), 11-30.

Examples

set.seed(123)
n <- 200
data <- data.frame(
  target_present = rep(c(TRUE, FALSE), each = n / 2),
  identification = sample(c("suspect", "filler", "reject"), n,
                          replace = TRUE, prob = c(0.5, 0.25, 0.25)),
  confidence = round(runif(n, 0, 100))
)
cal <- make_calibration_data(data, confidence_bins = c(0, 60, 80, 100))
cal$C
cal$calibration_data


r4lineups documentation built on Nov. 10, 2026, 1:08 a.m.