pcd2: Pseudo-count D2 method

In irtQ: Unidimensional Item Response Theory Modeling

Pseudo-count D2 method

Description

This function calculates the Pseudo-count D^{2} statistic to evaluate item parameter drift, as described by Cappaert et al. (2018) and Stone (2000). The Pseudo-count D^{2} statistic is designed to detect item parameter drift efficiently without requiring item recalibration, making it especially valuable in computerized adaptive testing (CAT) environments. This method compares observed and expected response frequencies across quadrature points, which represent latent ability levels. The expected frequencies are computed using the posterior distribution of each examinee's ability (Stone, 2000), providing a robust and sensitive measure of item parameter drift, ensuring the stability and accuracy of the test over time.

Usage

pcd2(
  x,
  data,
  D = 1,
  item.skip = NULL,
  missing = NA,
  Quadrature = c(49, 6),
  weights = NULL,
  group.mean = 0,
  group.var = 1,
  crit.val = NULL,
  min.resp = NULL,
  purify = FALSE,
  max.iter = 10,
  verbose = TRUE
)

Arguments

x	A data frame containing item metadata (e.g., item parameters, number of categories, IRT model types, etc.). See est_irt() or simdat() for more details about the item metadata. This data frame can be easily created using the shape_df() function.
data	A matrix of examinees' item responses corresponding to the items specified in the x argument. Rows represent examinees and columns represent items.
D	A scaling constant used in IRT models to make the logistic function closely approximate the normal ogive function. A value of 1.7 is commonly used for this purpose. Default is 1.
item.skip	A numeric vector of item indices to exclude from IPD analysis. If NULL, all items are included. Useful for omitting specific items based on prior insights.
missing	A value indicating missing responses in the data set. Default is NA.
Quadrature	A numeric vector of length two: first element: number of quadrature points second element: symmetric bound (absolute value) for those points For example, c(49, 6) specifies 49 evenly spaced points from -6 to 6. These points are used in the E-step of the EM algorithm. Default is c(49, 6).
weights	A two-column matrix or data frame containing the quadrature points (in the first column) and their corresponding weights (in the second column) for the latent variable prior distribution. If not NULL, the scale of the latent ability distribution is fixed to match the scale of the provided quadrature points and weights. The weights and points can be conveniently generated using the function gen.weight(). If NULL, a normal prior density is used instead, based on the information provided in the Quadrature, group.mean, and group.var arguments. Default is NULL.
group.mean	A numeric value specifying the mean of the latent variable prior distribution when weights = NULL. Default is 0. This value is fixed to resolve the indeterminacy of the item parameter scale during calibration.
group.var	A positive numeric value specifying the variance of the latent variable prior distribution when weights = NULL. Default is 1. This value is fixed to resolve the indeterminacy of the item parameter scale during calibration.
crit.val	A critical value applied in hypothesis testing using the Pseudo-count D^{2} statistic. Default is NULL.
min.resp	A positive integer specifying the minimum required number of responses for each evaluated item. Defaults to NULL.
purify	Logical. Indicates whether to apply a purification procedure. Default is FALSE.
max.iter	A positive integer specifying the maximum number of iterations allowed for the purification process. Default is 10.
verbose	Logical. If TRUE, progress messages from the purification procedure will be displayed; if FALSE, the messages will be suppressed. Default is TRUE.

Details

The Pseudo-count D^{2} statistic quantifies item parameter drift (IPD) by computing the weighted squared differences between the observed and expected response frequencies for each score category across ability levels. The expected frequencies are determined using the posterior distribution of each examinee's ability (Stone, 2000).

The Pseudo-count D^{2} statistic is calculated as:

Pseudo-count D^{2} = \sum_{k=1}^{Q} \left( \frac{r_{0k} + r_{1k}}{N}\right) \left( \frac{r_{1k}}{r_{0k} + r_{1k}} - E_{1k} \right)^2

where r_{0k} and r_{1k} are the pseudo-counts for the incorrect and correct responses at each ability level k, E_{1k} is the expected proportion of correct responses at each ability level k, calculated using item parameters from the item bank, and N is the total count of examinees who received each item.

Critical Value (crit.val): The crit.val argument specifies the threshold used to flag an item as exhibiting potential parameter drift. If an item's Pseudo-count D^{2} value exceeds this threshold, it is identified as a drifted item. If crit.val = NULL, the function reports the raw statistic without flagging.

Minimum Response Count (min.resp): The min.resp argument sets a lower bound on the number of responses required for an item to be included in the analysis. Items with fewer responses than min.resp are automatically excluded by replacing all their responses with NA. This avoids unreliable estimates based on small sample sizes.

Purification Procedure: Although Cappaert et al. (2018) did not incorporate purification into their method, pcd2() implements an optional iterative purification process similar to Lim et al. (2022). When purify = TRUE and a crit.val is provided:

• The procedure begins by identifying items flagged for drift using the initial Pseudo-count D^{2} statistics.

• In each subsequent iteration, the item with the highest flagged Pseudo-count D^{2} value is removed from the item set, and the statistics are recalculated using only the remaining items.

• The process continues until no additional items are flagged or the number of iterations reaches max.iter.

• All flagged items and statistics are saved, and convergence status is reported.

This process ensures that drift detection is not distorted by already-flagged items, improving the robustness of the results.

Value

This function returns a list containing four main components:

no_purify	A list containing the results of Pseudo-count D^{2} analysis without applying the purification procedure. It includes: ipd_stat A data frame summarizing the Pseudo-count D^{2} statistics for all items. The columns include: id (item ID), pcd2 (the computed D^{2} value), and N (the number of valid examinee responses per item). ipd_item A numeric vector of item indices that were flagged as exhibiting item parameter drift (IPD), based on the specified critical value crit.val. If no items are flagged or crit.val = NULL, this is NULL.
purify	A logical value indicating whether the iterative purification procedure was applied (TRUE) or not (FALSE).
with_purify	A list containing the results of Pseudo-count D^{2} analysis after applying the purification procedure. This list is populated only when both purify = TRUE and crit.val is not NULL. It includes: ipd_stat A data frame reporting the final Pseudo-count D^{2} statistics after purification. Columns include: id (item ID), pcd2 (the computed D^{2} value), N (the number of valid responses), and n.iter (the iteration number in which each item was evaluated). ipd_item A numeric vector of item indices flagged as IPD items during purification. Items are ordered by the iteration in which they were flagged. n.iter An integer indicating the number of purification iterations completed. complete A logical value indicating whether the purification procedure converged before reaching the maximum number of iterations (max.iter). If FALSE, the iteration limit was reached before convergence.
crit.val	A numeric value indicating the critical threshold used to flag items for parameter drift. If not specified by the user, this will be NULL.

Author(s)

Hwanggyu Lim [e-mail]

References

Cappaert, K. J., Wen, Y., & Chang, Y. F. (2018). Evaluating CAT-adjusted approaches for suspected item parameter drift detection. Measurement: Interdisciplinary Research and Perspectives, 16(4), 226-238.

Stone, C. A. (2000). Monte Carlo based null distribution for an alternative goodness-of-fit test statistic in IRT models. Journal of educational measurement, 37(1), 58-75.

Examples

## Example 1: No critical value specified
## Compute the Pseudo-count D2 statistics for dichotomous items
## Import the "-prm.txt" output file generated by flexMIRT
flex_sam <- system.file("extdata", "flexmirt_sample-prm.txt", package = "irtQ")

# Extract metadata for the first 30 3PLM items
x <- bring.flexmirt(file = flex_sam, "par")$Group1$full_df[1:30, 1:6]

# Generate abilities for 500 examinees from N(0, 1)
set.seed(25)
score <- rnorm(500, mean = 0, sd = 1)

# Simulate response data using the item metadata and ability values
data <- simdat(x = x, theta = score, D = 1)

# Compute the Pseudo-count D2 statistics (no purification applied)
ps_d2 <- pcd2(x = x, data = data)
print(ps_d2)

## Example 2: Applying a critical value with purification
# Compute the Pseudo-count D2 statistics with purification enabled
ps_d2_puri <- pcd2(x = x, data = data, crit.val = 0.002, purify = TRUE)
print(ps_d2_puri)


## -- Example 3: CAT-based IPD detection using simIPD --------------------------
##
## The Pseudo-count D2 statistic has no closed-form null distribution.
## Following Lim & Han (in press), the critical value is estimated empirically
## via bootstrap:
##   (1) Select drift-free (anchor) items to form the null D2 distribution.
##   (2) Repeatedly resample from those values and take the 95th percentile.
##   (3) Average across bootstrap iterations to obtain the critical value.
##
## The simIPD dataset provides:
##   item_par  : original 360-item pool (3PLM, irtQ format)
##   foc_resp  : focal group CAT responses (N = 3,000; TL = 30; ~92% NA)
##   key_item  : indices of 90 highly-exposed key items (IPD analysis targets)
##   item.skip : indices of 270 non-key items (excluded from analysis)
##   ipd_item  : 18 truly drifted items (ground truth; a & b each -0.5)
## -----------------------------------------------------------------------------

data(simIPD)

## -- Step 1. Select anchor items for bootstrap -----------------------------
## From the full item pool, select items that (a) have sufficient observed
## responses (>= boot_size) and (b) are not known IPD items. These items
## serve as the empirical null distribution of D2.
## boot_size = 300: chosen to match the minimum response count used in the
##   bootstrap procedure of the simulation study (Lim & Han, in press).
## In practice, exclude items you know or suspect have drifted; here the
## ground truth (simIPD$ipd_item) is used for illustration.
boot_size    <- 300
n_resp       <- colSums(!is.na(simIPD$foc_resp))   # observed responses per item
anchor_items <- setdiff(which(n_resp >= boot_size), simIPD$ipd_item)

## -- Step 2. Compute null D2 values for the anchor items ------------------
pcd2_null <- pcd2(
  x        = simIPD$item_par[anchor_items, ],
  data     = simIPD$foc_resp[, anchor_items],
  D        = 1.7,
  crit.val = NULL,
  purify   = FALSE
)$no_purify$ipd_stat$pcd2

## -- Step 3. Bootstrap critical value (Lim & Han, in press) --------------
## For each bootstrap iteration: resample boot_size D2 values from the null
## distribution and take the 95th percentile. The critical value is the mean
## of these percentiles across all iterations.
## n_boots = 500: reduced for demo speed; use 10,000 in practice for a
##   stable critical value estimate.
set.seed(2024)
n_boots <- 500

crit_val <- purrr::map_dbl(
  .x = 1:n_boots,
  .f = ~ sample(x = pcd2_null, size = boot_size, replace = TRUE) |>
           stats::quantile(probs = 0.95)
) |> mean()

cat("Bootstrap critical value (alpha = 0.05):", round(crit_val, 6), "\n")

## -- Step 4. Run PCD2 with the bootstrap critical value + purification --
pcd2_result <- pcd2(
  x         = simIPD$item_par,
  data      = simIPD$foc_resp,
  D         = 1.7,
  item.skip = simIPD$item.skip,
  crit.val  = crit_val,
  purify    = TRUE,
  max.iter  = 30
)

## -- Step 5. Compare detected items to ground truth --------------------
detected <- pcd2_result$with_purify$ipd_item
cat("Truly drifted items (ground truth):", simIPD$ipd_item, "\n")
cat("PCD2-detected items:               ", detected, "\n")
cat("True positives:", sum(detected %in% simIPD$ipd_item), "of",
    length(simIPD$ipd_item), "\n")

irtQ documentation built on July 27, 2026, 9:08 a.m.