cac_rud: Classification Accuracy and Consistency Based on Rudner's...

In irtQ: Unidimensional Item Response Theory Modeling

Classification Accuracy and Consistency Based on Rudner's (2001, 2005) Approach

Description

This function computes classification accuracy and consistency indices using the method proposed by Rudner in 2001 and 2005. This function supports both scenarios: when the empirical ability distribution of the population is available, and when individual ability estimates are used.

Usage

cac_rud(x = NULL, cutscore, theta = NULL, se = NULL, weights = NULL, D = 1)

Arguments

x	A data frame containing item metadata (e.g., item parameters, number of categories, IRT model types, etc.). See est_irt() or simdat() for more details about the item metadata. This data frame can be easily created using the shape_df() function. If x = NULL, the se argument must be explicitly provided. Defaults to NULL.
cutscore	A numeric vector specifying the cut scores for classification. Cut scores are the points that separate different performance categories (e.g., pass vs. fail, or different grades).
theta	A numeric vector of ability estimates. Ability estimates (theta values) are the individual proficiency estimates obtained from the IRT model. The theta parameter is optional and can be NULL.
se	A numeric vector of the same length as theta representing the standard errors associated with each ability estimate. If NULL and x is supplied, standard errors are computed using the test information function. See the Details section for more information
weights	An optional two-column data frame or matrix where the first column is the quadrature points (nodes) and the second column is the corresponding weights. This is typically used in quadrature-based IRT analysis.
D	A scaling constant used in IRT models to make the logistic function closely approximate the normal ogive function. A value of 1.7 is commonly used for this purpose. Default is 1.

Details

This function first validates the input arguments. If both theta and weights are NULL, the function will stop and return an error message. Either theta or weights must be specified.

Either x or se must be specified. If se is not provided (i.e., se = NULL), it will be computed using the test information derived from the item metadata x. The length of se must match the length of theta, or the number of quadrature points in weights.

It then computes the probability that an examinee with a given ability is classified into each performance level using the normal distribution function centered at each theta (or quadrature point) with standard deviation se. These probabilities are used to calculate conditional classification accuracy (the probability of being correctly classified) and conditional classification consistency (the probability of being consistently classified upon repeated testing) for each ability value.

Finally, the function computes marginal classification accuracy and consistency across all examinees by aggregating the conditional indices with the associated weights.

Value

A list containing the following elements:

• confusion: A confusion matrix showing the cross table between true and expected levels.

• marginal: A data frame showing the marginal classification accuracy and consistency indices.

• conditional: A data frame showing the conditional classification accuracy and consistency indices.

• prob.level: A data frame showing the probability of being assigned to each level category.

• cutscore: A numeric vector showing the cut scores used in the analysis.

Author(s)

Hwanggyu Lim [e-mail]

References

Rudner, L. M. (2001). Computing the expected proportions of misclassified examinees. Practical Assessment, Research, and Evaluation, 7(1), 14.

Rudner, L. M. (2005). Expected classification accuracy. Practical Assessment, Research, and Evaluation, 10(1), 13.

See Also

gen.weight(), est_score(), cac_lee()

Examples

## -------------------------------------------
# 1. Using the empirical ability distribution
## -------------------------------------------

# Import the "-prm.txt" output file from flexMIRT
flex_prm <- system.file("extdata", "flexmirt_sample-prm.txt", package = "irtQ")

# Read item parameter estimates and convert them into item metadata
x <- bring.flexmirt(file = flex_prm, "par")$Group1$full_df

# Define cut scores on the theta scale
cutscore <- c(-2, -0.5, 0.8)

# Create quadrature points and corresponding weights
node <- seq(-4, 4, 0.25)
weights <- gen.weight(dist = "norm", mu = 0, sigma = 1, theta = node)

# Compute classification accuracy and consistency
cac_1 <- cac_rud(
  x = x, 
  cutscore = cutscore, 
  weights = weights,
  se = NULL,  
  D = 1)
print(cac_1)

## -----------------------------------------
# 2. Using individual ability estimates
## -----------------------------------------

# Generate true abilities from N(0, 1)
set.seed(12)
theta <- rnorm(n = 1000, mean = 0, sd = 1)

# Simulate item response data
data <- simdat(x = x, theta = theta, D = 1)

# Estimate ability and standard errors using ML estimation
est_theta <- est_score(
  x = x, data = data, D = 1, method = "ML",
  range = c(-4, 4), se = TRUE
)
theta_hat <- est_theta$est.theta
se <- est_theta$se.theta

# Compute classification accuracy and consistency using provided SEs
cac_2 <- cac_rud(
  cutscore = cutscore, 
  theta = theta_hat, 
  se = se)
print(cac_2)

# Or compute classification accuracy and consistency using the item metadata
# instead of providing the SEs directly
cac_2 <- cac_rud(
  x = x, 
  cutscore = cutscore, 
  theta = theta_hat)
print(cac_2)

irtQ documentation built on July 27, 2026, 9:08 a.m.