pcd2: Pseudo-count D2 method

In irtQ: Unidimensional Item Response Theory Modeling

Pseudo-count D2 method

Description

This function calculates the Pseudo-count D^{2} statistic to evaluate item parameter drift, as described by Cappaert et al. (2018) and Stone (2000). The Pseudo-count D^{2} statistic is designed to detect item parameter drift efficiently without requiring item recalibration, making it especially valuable in computerized adaptive testing (CAT) environments. This method compares observed and expected response frequencies across quadrature points, which represent latent ability levels. The expected frequencies are computed using the posterior distribution of each examinee's ability (Stone, 2000), providing a robust and sensitive measure of item parameter drift, ensuring the stability and accuracy of the test over time.

Usage

pcd2(
  x,
  data,
  D = 1,
  missing = NA,
  Quadrature = c(49, 6),
  weights = NULL,
  group.mean = 0,
  group.var = 1
)

Arguments

x	A data frame containing the metadata for the item bank, which includes important information for each item such as the number of score categories and the IRT model applied. See est_irt, irtfit, info or simdat for more detail about the item metadata.
data	A matrix containing examinees' response data of the items in the argument x. A row and column indicate the examinees and items, respectively.
D	A scaling factor in IRT models to make the logistic function as close as possible to the normal ogive function (if set to 1.7). Default is 1.
missing	A value indicating missing values in the response data set. Default is NA.
Quadrature	A numeric vector of two components specifying the number of quadrature points (in the first component) and the symmetric minimum and maximum values of these points (in the second component). For example, a vector of c(49, 6) indicates 49 rectangular quadrature points over -6 and 6.
weights	A two-column matrix or data frame containing the quadrature points (in the first column) and the corresponding weights (in the second column) of the latent variable prior distribution. If not NULL, the scale of the latent ability distribution will be will be fixed to the scale of the provided quadrature points and weights. The weights and quadrature points can be easily obtained using the function gen.weight. If NULL, a normal prior density is used based on the information provided in the arguments of Quadrature, group.mean, and group.var). Default is NULL.
group.mean	A numeric value to set the mean of latent variable prior distribution when weights = NULL. Default is 0.
group.var	A positive numeric value to set the variance of latent variable prior distribution when weights = NULL. Default is 1.

Details

The Pseudo-count D^{2} values are calculated by summing the weighted squared differences between the observed and expected frequencies for each score category across all items. The expected frequencies are determined using the posterior distribution of each examinee's ability (Stone, 2000).

The Pseudo-count D^{2} statistic is calculated as:

Pseudo-count D^{2} = \sum_{k=1}^{Q} \left( \frac{r_{0k} + r_{1k}}{N}\right) \left( \frac{r_{1k}}{r_{0k} + r_{1k}} - E_{1k} \right)^2

where r_{0k} and r_{1k} are the pseudo-counts for the incorrect and correct responses at each ability level k, E_{1k} is the expected proportion of correct responses at each ability level k, calculated using item parameters from the item bank, and N is the total count of examinees who received each item

The pcd2 function is designed to be flexible and allows for detailed control over the computation process through its various arguments:

x

The metadata should include key information such as the number of score categories for each item, the IRT model applied (e.g., "1PLM", "2PLM", "3PLM"), and the item parameters (e.g., discrimination, difficulty, guessing). This data frame is crucial because it defines the structure of the items and the parameters used in the calculation of expected frequencies.

data

The response matrix should be preprocessed to ensure that missing values are correctly coded (using the 'missing' argument if needed). The matrix is the foundation for calculating both observed and expected frequencies for each item and score category. Ensure that the number of items in this matrix matches the number specified in the 'x' argument.

Quadrature

The quadrature points are used to approximate the latent ability distribution, which is essential for accurately estimating the posterior distribution of abilities. Adjust the number and range of quadrature points based on the precision required and the characteristics of the examinee population.

weights

This argument allows you to provide a custom two-column matrix or data frame where the first column contains quadrature points and the second column contains the corresponding weights. This provides flexibility in defining the latent ability distribution. If 'weights' is set to 'NULL', the function generates a normal prior distribution based on the values provided in 'Quadrature', 'group.mean', and 'group.var'. Use this argument when you have a specific latent ability distribution you wish to apply, such as one derived from empirical data or an alternative theoretical distribution.

group.mean, group.var

These numeric values define the mean and variance of the latent ability distribution when 'weights' is not provided. The default values are 'group.mean = 0' and 'group.var = 1', which assume a standard normal distribution. These values are used to generate quadrature points and weights internally if 'weights' is 'NULL'. Adjust 'group.mean' and 'group.var' to reflect the expected distribution of abilities in your examinee population, particularly if you suspect that the latent trait distribution deviates from normality.

When setting these arguments, consider the specific characteristics of your item bank, the distribution of abilities in your examinee population, and the computational precision required. Properly configured, the function provides robust and accurate Pseudo-count D^{2} statistics, enabling effective monitoring of item parameter drift in CAT or other IRT-based testing environments.

Value

A data frame containing the Pseudo-count D^{2} statistic for each item in the analysis. The data frame includes the following columns:

id	The unique identifier for each item, corresponding to the item IDs provided in the x argument.
pcd2	The computed Pseudo-count D^{2} statistic for each item, which quantifies the degree of item parameter drift by comparing observed and expected response frequencies.
N	The number of examinees whose responses were used to calculate the Pseudo-count D^{2} statistic for each item, reflecting the sample size involved in the computation.

Author(s)

Hwanggyu Lim [e-mail]

References

Cappaert, K. J., Wen, Y., & Chang, Y. F. (2018). Evaluating CAT-adjusted approaches for suspected item parameter drift detection. Measurement: Interdisciplinary Research and Perspectives, 16(4), 226-238.

Stone, C. A. (2000). Monte Carlo based null distribution for an alternative goodness-of-fit test statistic in IRT models. Journal of educational measurement, 37(1), 58-75.

Examples

## Compute the pseudo-count D2 statistics for the dichotomous items
## import the "-prm.txt" output file from flexMIRT
flex_sam <- system.file("extdata", "flexmirt_sample-prm.txt", package = "irtQ")

# select the first 30 3PLM item metadata to be examined
x <- bring.flexmirt(file = flex_sam, "par")$Group1$full_df[1:30, 1:6]

# generate examinees' abilities from N(0, 1)
set.seed(25)
score <- rnorm(500, mean = 0, sd = 1)

# simulate the response data
data <- simdat(x = x, theta = score, D = 1)

# compute the pseudo-count D2 statistics
ps_d2 <- pcd2(x = x, data = data)
print(ps_d2)

irtQ documentation built on Sept. 11, 2024, 5:10 p.m.