View source: R/calibration_functions.R
| make_calibration_data | R Documentation |
Computes calibration analysis metrics following Juslin, Olsson, & Winman (1996). Calibration analysis assesses the match between confidence and accuracy, providing a more appropriate measure than simple confidence-accuracy correlation.
make_calibration_data(
data,
confidence_bins = NULL,
choosers_only = TRUE,
lineup_size = 6,
confidence_scale = c("auto", "0-1", "0-100"),
innocent_suspect = "auto",
population = NULL,
tp_fillers = c("exclude", "error")
)
data |
A dataframe with the following columns:
|
confidence_bins |
Numeric vector of bin edges for grouping confidence (e.g., c(0, 60, 80, 100) creates bins 0-60, 60-80, 80-100). If NULL, uses individual confidence levels. |
choosers_only |
Logical. If TRUE (default), only analyze suspect identifications
( |
lineup_size |
Integer. Number of people in lineup (default = 6). Used for estimating incorrect suspect IDs from filler choices in target-absent lineups. |
confidence_scale |
How the confidence scale is interpreted: "auto" (default; max <= 1 treated as 0-1, otherwise as 0-100, with a warning for ambiguous Likert/0-10 scales), "0-1", or "0-100". C and O/U require a 0-1/0-100 probability scale. |
innocent_suspect |
How innocent-suspect identifications in
target-absent lineups are obtained. |
population |
Which responses are calibrated: |
tp_fillers |
For |
Calibration analysis distinguishes between:
Calibration (C): How well confidence matches accuracy (perfect = 0)
Over/underconfidence (O/U): Overall bias in confidence judgments (0 = no net bias in the mean; overconfident and underconfident bins can cancel, so perfect calibration needs C = 0)
Resolution (NRI): Ability to discriminate correct from incorrect responses
The calibration statistic C measures the weighted mean squared deviation between mean confidence and accuracy in each bin:
C = \sum_{j=1}^{J} \frac{n_j}{N} (c_j - a_j)^2
The Normalized Resolution Index (NRI) captures how well confidence discriminates correct from incorrect responses:
NRI = \frac{\frac{1}{N} \sum_{j=1}^{J} n_j (a_j - \bar{a})^2}{\bar{a}(1-\bar{a})}
The default population, suspect identifications, is the one used in CAC
analysis (Mickes, 2015). In the eyewitness calibration literature (e.g.,
Brewer & Wells, 2006) choosers are witnesses who identify someone. Brewer and
Wells excluded filler identifications from target-present lineups, because a
filler is known not to be the culprit, and scored all target-absent
identifications as errors; population = "choosers" reproduces that
analysis, and tp_fillers = "error" keeps the target-present filler
identifications as errors instead. Nonchoosers (population = "nonchoosers")
are witnesses who reject the lineup; a rejection is correct when the culprit
is absent.
If target-absent data contain designated innocent-suspect identifications,
those observations are used directly. Otherwise, each target-absent filler
identification contributes weight 1 / lineup_size as an estimate of
innocent-suspect identifications. In that latter case the per-bin counts and
effective total used by C, O/U, NRI, and downstream ANRI can be fractional;
the results are lineup-size approximations rather than statistics from an
observed designated-suspect outcome.
A list containing:
calibration_data: Dataframe with per-bin confidence, accuracy, and counts
C: Calibration statistic (weighted mean squared difference between confidence and accuracy)
OU: Over/underconfidence (mean confidence minus mean accuracy)
NRI: Normalized Resolution Index, the between-bin variance of accuracy divided by \bar{a}(1 - \bar{a}), the total variance of the outcome (the proportion of outcome variance explained by the confidence bins)
overall_accuracy: Mean accuracy across all responses
overall_confidence: Mean confidence across all responses
n_total: Total number of responses analyzed
Mickes, L. (2015). Receiver operating characteristic analysis and confidence-accuracy characteristic analysis in investigations of system variables and estimator variables that affect eyewitness memory. Journal of Applied Research in Memory and Cognition, 4(2), 93-102.
Juslin, P., Olsson, N., & Winman, A. (1996). Calibration and diagnosticity of confidence in eyewitness identification: Comments on what can be inferred from the low confidence-accuracy correlation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22(5), 1304-1316.
Brewer, N., & Wells, G. L. (2006). The confidence-accuracy relationship in eyewitness identification: Effects of lineup instructions, foil similarity, and target-absent base rates. Journal of Experimental Psychology: Applied, 12(1), 11-30.
set.seed(123)
n <- 200
data <- data.frame(
target_present = rep(c(TRUE, FALSE), each = n / 2),
identification = sample(c("suspect", "filler", "reject"), n,
replace = TRUE, prob = c(0.5, 0.25, 0.25)),
confidence = round(runif(n, 0, 100))
)
cal <- make_calibration_data(data, confidence_bins = c(0, 60, 80, 100))
cal$C
cal$calibration_data
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.