simulateRespondents: Simulation of multiple examinees of adaptive tests

In catR: Generation of IRT Response Patterns under Computerized Adaptive Testing

Simulation of multiple examinees of adaptive tests

Description

This command runs a set of adaptive tests, for a given item bank, a set of ability levels, a possible matrix of item responses, and several lists of CAT parameters (starting items, stopping rule, provisional and final ability estimators).

Usage

simulateRespondents(thetas, itemBank, responsesMatrix = NULL, model = NULL, 
  	genSeed = NULL, cbControl = NULL, rmax = 1, Mrmax = "restricted", 
  	start = list(fixItems = NULL, seed = NULL, nrItems = 1, theta = 0, 
  	D = 1, randomesque = 1, random.seed = NULL, startSelect = "MFI", 
  	cb.control = FALSE, random.cb = NULL), test = list(method = "BM", 
  	priorDist = "norm", priorPar = c(0,1), weight = "Huber", tuCo = 1, 
  	sem.type = "classic", sem.exact = FALSE, se.ase = 10, range = c(-4, 4), 
  	D = 1, parInt = c(-4, 4, 33), itemSelect = "MFI", infoType = "observed", 
  	randomesque = 1, random.seed = NULL, AP = 1, proRule = "length", 
  	proThr = 20, constantPatt = NULL), stop = list(rule = "length", 
  	thr = 20, alpha = 0.05), final = list(method = "BM", priorDist = "norm", 
  	priorPar = c(0,1), weight = "Huber", tuCo = 1, sem.type = "classic", 
  	sem.exact = FALSE, range = c(-4, 4), D = 1, parInt = c(-4, 4, 33), 
  	alpha = 0.05), save.output = FALSE, output = c("", "catR", "csv"))
## S3 method for class 'catResult'
print(x, ...)
## S3 method for class 'catResult'
plot(x, type = "all", deciles = "theta", save.plot = FALSE, 
  	save.options = c("", "plot", "pdf"), res = 300, ...)
 

Arguments

thetas	numeric: a vector of true ability values for which a CAT must be generated for each component.
itemBank	numeric: a suitable matrix of item parameters (possibly augmented by group membership for content balancing). See Details.
responsesMatrix	numeric: either NULL (default) or a suitable matrix of item responses. See Details.
model	either NULL (default) for dichotomous models, or any suitable acronym for polytomous models. Possible values are "GRM", "MGRM", "PCM", "GPCM", "RSM" and "NRM". See Details.
genSeed	either a vector of numeric values to fix the random seed of each generated pattern, or NULL (default). Ignored if responsesMatrix is not NULL. See Details.
cbControl	either a list of accurate format to control for content balancing, or NULL. See Details.
rmax	numeric: the maximum exposure rate (default is 1).
Mrmax	character: the method for controlling maximum exposure rate. Possible values are "restricted"(default) or "IE". See Details.
start	a list with the options for starting the adaptive test. See Details.
test	a list with the options for provisional ability estimation and next item selection. See Details.
stop	a list with the options of the stopping rule. See Details.
final	a list with the options for final ability estimation. See Details.
save.output	logical: should the output be saved in an external text file? (default is FALSE).
output	character: a vector of three components. The first component is either the file path to save the output or "" (default), the second component is either the initial part of the name of the output file or "catR", and the third component is the file type, either "txt" or "csv" (default). See Details.
x	an object of class "cat", typically an output of simulateRespondents function.
type	character: the type of plot to display. Possible values are "all" (default), "trueEst", "expRate", "cumExpRate", "cumNumberItems", "expRatePara", "condBias", "condRMSE", "numberItems", "sError" and ="condThr". See Details.
deciles	whether the deciles number ("deciles") or the mean ability level per decile ("theta") will be used in the axis of the plots.
save.plot	logical: should the plot be saved in an external figure? (default is FALSE).
save.options	character: a vector of three components. The first component is either the file path or "" (default), the second component is the name of the output file or ,"plot" (default), and the third component is the file extension, either "pdf" (default) or "jpeg". Ignored if save.plot is FALSE. See Details.
res	numeric: the resolution for JPEG figures (default value is 300).
...	other generic arguments to be passed to print and plot functions.

Details

The simulateRespondents function permits to generate several adaptive tests to a set of respondents defined by their ability levels. It makes a repeated call to an adaptive test using an item bank specified by arguments itemBank model, and with the same start, test, stop and final lists. Content balancing can also be controlled for each respondent with the cbControl argument. All arguments of simulateRespondents are used in exactly the same manner as in randomCAT (so refer to this function for further information), except the following four.

First, thetas is now a vector of ability levels, and a CAT will be generated for each component of thetas. If responsesMatrix is NULL, item responses are generated from the IRT model, item bank parameters, and the ability levels. In this case, thetas can be considered as real ability levels. Otherwise, responsesMatrix must be provided as a matrix with as many rows as the length of thetas and as many columns as the number of items in itemBank. Each row contains the response pattern of one examinee whose ability level is given by the value of the corresponding component of thetas. Note that only allowable item responses can be included in responsesMatrix and missing values are not accepted. Fixing the random seed can also be done with the genSeed argument. The latter must hold as many components as the vector thetas, otherwise an error message is returned. Each component of genSeed is used to fix the seed for ach pattern generation.

The option of providing a response matrix through responsesMatrix is considered for two possible uses:

1. post-hoc simulations: examinees provided responses to the full item bank and one wants to test the performance of a CAT with those responses,

2. simulations considering item parameter estimation errors: responses to the full bank are generated with the real parameters and the CAT is run is the estimated parameters and the responses from the correct model.

Note that if thetas holds a single value, then the function simply calls randomCAT and returns its output instead.

Second, rmax fixes the desired maximum exposure rate for all items. Default value is 1, allowing thus items to be administered to all respondents without restrictions.

Third, the Mrmax argument fixes the method to constraint exposure rates to be smaller than the maximum allowed rate. Possible methods are the restricted method ("restricted"; Revuelta and Ponsoda, 1998) and the item-eligibility method ("IE"; van der Linden and Veldkamp, 2004). A description of both methods can be found in Barrada, Abad and Veldkamp (2009)

Fourth, if the length of thetas is greater than 1, save.output for randomCAT is fixed to "FALSE". Otherwise, the same file would be overwritten for each new respondent.

The output of simulateRespondents, as displayed by the print.catResult function, proposes summary statistics related to overall accuracy (bias, RMSE, etc), conditional accuracy per decile, and item exposure control (minimum and maximum exposure rates, test overlap rate, etc) among all generated CATs. This output can be saved when save.output is set to TRUE, and if so, three outpout files are returned: (1) one with the main summary statistics, with overall results and conditional on decile results; (2) the file with respondents' patterns, items administered and provisional ability estimates; and (3) a table with true and estimated ability levels, final standard errors and numbers of items administered per respondent. Specific information provided and saved depends on the rule used in stop.

This output can also be graphically displayed with the plot.catResult function. In addition to the output of the function and the item bank, it takes the argument type to determine which plot should be returned. Ten different single plots can be displayed:

1. "trueEst": the scatterplot of true vs. estimated ability levels.

2. "expRate": the exposure rates of the items, having ranked them according to these exposure rates.

3. "cumExpRate": the cumulative exposure rates of the items, having ranked them according to these exposure rates.

4. "cumNumberItems": the test length as a function of cumulative percent of examinees. This plot is not available when rule is 'length'.

5. "expRatePara": the scatterplot of item exposure rates vs. item discrimination parameters. This plot is not available when model is 'PCM' or 'NRM', as in those IRT models there are no discrimination parameters.

6. "condBias": the conditional bias of ability estimation as a function of the deciles of the true ability levels.

7. "condRMSE": the conditional RMSE of ability estimation as a function of the deciles of the true ability levels.

8. "numberItems": the conditional test length as a function of the deciles of the true ability levels. This plot is not available when rule is 'length'.

9. "sError": the conditional standard error of ability estimation as a function of the deciles of the true ability levels.

10. "condThr": the conditional proportions of CATs satisfying the 'precision' or 'classification' stopping rule, as a function of the deciles of the true ability levels. This plot is not available when rule is 'length'.

In addition, the value "all" (default value) displays several available plots in a single panel. Displayed plots depend on the rule used in stop: (a) with 'length', "trueEst", "condBias", "condRMSE", "expRate", "cumExpRate", "expRatePara"; (b) with 'precision' and 'classification', all the plots but "cumExpRate".

These plots can be saved as external PDF or JPEG files, by setting save.plot to TRUE and defining the arguments of save.options accurately (see randomCAT for further explanations and the Examples section below).

Value

The function simulateRespondents returns a list of class "catResult" with the following arguments:

thetas	the value of the thetas argument.
itemBank	the value of the itemBank argument.
responsesMatrix	the value of the responsesMatrix argument.
model	the value of the model argument.
genSeed	the value of the genSeed argument.
cbControl	the value of the cbControl argument.
rmax	the value of the rmax argument.
Mrmax	the value of the Mrmax argument.
start	the value of the start argument.
test	the value of the test argument.
stop	the value of the stop argument.
final	the value of the final argument.
save.output	the value of the save.output argument.
output	the value of the output argument.
estimatedThetas	a vector with (final) estimated ability levels.
correlation	the correlation between the thetas vector and estimated ability levels.
bias	the value of the bias between true and estimated ability levels.
RMSE	the value of the RMSE between true and estimated ability levels.
thrOK	a vector indicating whether the respondents finished the test satisfying (1) or not (0) the stop criteria.
exposureRates	a vector with empirical exposure rates of all item in the bank.
testLength	the mean test length.
overlap	the item overlap rate.
numberItems	a vector with the lengths of each adaptive test (i.e. the number of items administered).
condTheta	a vector with the mean ability level per decile.
condBias	a vector with conditional mean bias per decile.
condRMSE	a vector with conditional RMSE per decile.
condnItems	a vector with conditional mean test length per decile.
condSE	a vector with conditional mean standard error per decile.
condthrOK	a vector with conditional proportion of respondents that finish the test satisfying the stop criteria per decile.
ndecile	a vector with the number of respondents per decile.
final.values.df	a data frame with true ability levels, final ability estimates and standard errors, and test lengths.
responses.df	a data frame with all items administered, all item responses and all provisional ability estimates. -99 is displayed when the actual test length is smaller than the maximum test length for those item positions where no item was administered as the stop criterion was already reached.
start.time	the CPU time at the start of the CAT generation.
finish.time	the CPU time at the end of all CAT generations.

Author(s)

Juan Ramon Barrada
Department of Psychology and Sociology, Universidad Zaragoza, Spain
[e-mail]

David Magis
Department of Psychology, University of Liege, Belgium
[e-mail]

Guido Corradi
Department of Psychology and Sociology, Universidad Zaragoza, Spain
[e-mail]

References

Barrada, J. R., Abad, F. J., and Veldkamp, B. P. (2009). Comparison of methods for controlling maximum exposure rates in computerized adaptive testing. Psicothema, 21, 313-320.

Magis, D. and Barrada, J. R. (2017). Computerized Adaptive Testing with R: Recent Updates of the Package catR. Journal of Statistical Software, Code Snippets, 76(1), 1-18. doi: 10.18637/jss.v076.c01

Magis, D., and Raiche, G. (2012). Random Generation of Response Patterns under Computerized Adaptive Testing with the R Package catR. Journal of Statistical Software, 48 (8), 1-31. doi: 10.18637/jss.v048.i08

Revuelta, J., and Ponsoda, V. (1998). A comparison of item exposure control methods in computerized adaptive testing. Journal of Educational Measurement, 35, 311-327. doi: 10.1111/j.1745-3984.1998.tb00541.x

van der Linden, W.J., and Veldkamp, B.P. (2004). Constraining item exposure in computerized adaptive testing with shadow tests. Journal of Educational and Behavioral Statistics, 29, 273-291. doi: 10.3102/10769986029003273

See Also

randomCAT

Examples

## Dichotomous IRT model ##

 # Loading the 'tcals' parameters 
 data(tcals)
 bank <- as.matrix(tcals[,1:4])
 
 # Creation of a starting list with three theta values
 start <- list(theta = -1:1, randomesque = 5)

 # Creation of 'test' list: maximum likelihood estimation and
 # progressive method
 test <- list(method = "ML", itemSelect = "progressive")

 # Creation of a stopping rule: precision criterion, standard
 # error to be reached 0.3
 stop <- list(rule = "precision", thr = 0.3)

 # Creation of 'final' list: ML estimation of final ability
 final <- list(method = "ML")

 # Generation of ten respondents
 set.seed(1)
 thetas <- rnorm(10)

 # Default CAT generations, output not saved
 res <- simulateRespondents(thetas, bank, start = start, test = test, stop = stop, 
  final = final)

 # Maximum exposure restricted to 0.8
 res2 <- simulateRespondents(thetas, bank, start = start, test = test, stop = stop, 
  final = final, rmax = 0.8)

## Not run: 

 # Output saved
 res3 <- simulateRespondents(thetas, bank, start = start, test = test, stop = stop, 
  final = final, save.output = TRUE, output = c("C:/Program Files/", "out", "txt"))

## End(Not run)

 # With content balancing #

 # Creation of an appropriate list for content balancing
 # Equal proportions across subgroups of items
 cbList <- list(names = c("Audio1", "Audio2", "Written1", "Written2", "Written3"), 
        props = c(0.1, 0.2, 0.2, 0.2, 0.3))

 # CAT test (same options as above)
 res4 <- simulateRespondents(thetas, tcals, start = start, test = test, stop = stop, 
  final = final, cbControl = cbList)

 # Plotting and saving output #

 # Plotting all possible panels
 plot(res)
 plot(res, deciles = "deciles")

## Not run: 

 # Saving the plot in the "fig" pdf file in "c:/Program Files/"
 plot(res, save.plot = TRUE, save.options = c("c:/Program Files/", "fig", "pdf"))

## End(Not run)

 # Plotting the 'trueEst' type of plot
 plot(res, type = "trueEst")
 

catR documentation built on June 24, 2022, 9:06 a.m.