compare_pauc: Compare pAUC Between Two ROC Curves

View source: R/pauc_comparison.R

compare_paucR Documentation

Compare pAUC Between Two ROC Curves

Description

Statistically compares partial Area Under the Curve (pAUC) values between two lineup conditions using bootstrap-based standard errors and z-tests. Follows the methodology from pyWitness (Mickes et al., 2024).

Usage

compare_pauc(
  data1,
  data2,
  lineup_size = 6,
  max_false_id_rate = NULL,
  n_bootstrap = 2000,
  conf_level = 0.95,
  seed = NULL,
  label1 = "Condition 1",
  label2 = "Condition 2",
  innocent_suspect = "auto"
)

Arguments

data1

First dataset (dataframe with target_present, identification, confidence)

data2

Second dataset (same format as data1)

lineup_size

Integer. Number of people in lineup (default = 6)

max_false_id_rate

Numeric. Upper limit of the false-ID rate region over which partial AUC is computed. Default NULL uses the largest region both conditions actually reach, i.e. the smaller of their two maximum observed false-ID rates. Supplying a larger value means at least one condition is integrated only as far as its own data extend, so the two areas cover different intervals; a warning is issued in that case.

n_bootstrap

Integer. Number of bootstrap samples (default = 2000)

conf_level

Numeric. Confidence level for intervals (default = 0.95)

seed

Integer. Random seed for reproducibility (default = NULL)

label1

Character. Label for first condition (default = "Condition 1")

label2

Character. Label for second condition (default = "Condition 2")

innocent_suspect

How innocent-suspect identifications in target-absent lineups are obtained. "designated": the lineups had a designated innocent suspect, so target-absent "suspect" rows are counted directly, and zero such rows means a false-ID rate of zero. "estimated": there was no designated innocent suspect, so the innocent-suspect count is estimated as filler IDs divided by lineup_size. "auto" (the default) chooses "designated" when any target-absent row is a suspect ID or the data have a non-missing suspect_pos column, and "estimated" otherwise, with a message.

Details

This function implements an approximate statistical test for comparing ROC curves between two conditions (e.g., simultaneous vs. sequential lineups, different retention intervals, etc.).

**Method:**

  1. Computes pAUC for each condition (optionally up to a maximum false ID rate)

  2. Uses bootstrap resampling to estimate standard errors

  3. Computes Z-statistic: Z = (pAUC1 - pAUC2) / SE(pAUC1 - pAUC2)

  4. Calculates p-value from standard normal distribution

The two datasets are treated as independent conditions. This function does not implement a paired or clustered bootstrap.

**Interpretation:**

  • Positive pAUC difference: Condition 1 has the larger pAUC over the shared false-ID range

  • Negative pAUC difference: Condition 2 has the larger pAUC over the shared false-ID range

  • p < 0.05: Significant difference in pAUC between conditions; otherwise there is no significant difference

The comparison covers only the shared false-ID range and uses suspect IDs only, so it is not a general statement about discriminability. A common-range pAUC can mislead when a procedure raises both culprit and innocent-suspect IDs (Smith et al., 2019).

The independent-sample bootstrap estimates uncertainty in an empirical partial-area functional; the final comparison uses a normal/Wald interval. It is approximate and can be liberal in finite samples. A data-selected cutoff, sparse high-confidence cells, or resamples that do not reach the cutoff require additional caution. It is not an exact discrimination test.

Value

An object of class "pauc_comparison" containing:

  • pauc1: pAUC for condition 1

  • pauc2: pAUC for condition 2

  • pauc_diff: Difference in pAUC (pauc1 - pauc2)

  • se_pauc1: Bootstrap standard error for pAUC1

  • se_pauc2: Bootstrap standard error for pAUC2

  • se_diff: Standard error of the difference

  • z_score: Z-statistic for difference test

  • p_value: Two-tailed p-value

  • ci_diff: Confidence interval for difference

  • roc1: ROC object for condition 1

  • roc2: ROC object for condition 2

  • max_false_id_rate: Cutoff used for pAUC

  • label1, label2: Condition labels

  • n_bootstrap: Number of bootstrap samples used

References

Smith, A. M., Lampinen, J. M., Wells, G. L., Smalarz, L., & Mackovichova, S. (2019). Deviation from perfect performance measures the diagnostic utility of eyewitness lineups but partial area under the ROC curve does not. Journal of Applied Research in Memory and Cognition, 8(1), 50-59. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1016/j.jarmac.2018.09.003")}

Mickes, L., Seale-Carlisle, T. M., Chen, X., & Boogert, S. (2024). pyWitness 1.0: A python eyewitness identification analysis toolkit. Behavior Research Methods, 56, 1533-1550.

Wixted, J. T., & Mickes, L. (2012). The field of eyewitness memory should abandon probative value and embrace receiver operating characteristic analysis. Perspectives on Psychological Science, 7(3), 275-278.

Examples


# Compare two lineup procedures (halves of the example data)
data(lineup_example)
odd <- seq(1, nrow(lineup_example), by = 2)
comparison <- compare_pauc(
  lineup_example[odd, ],
  lineup_example[-odd, ],
  label1 = "Procedure A",
  label2 = "Procedure B",
  n_bootstrap = 200,
  seed = 123
)

print(comparison)
plot(comparison)



r4lineups documentation built on Nov. 10, 2026, 1:08 a.m.