itChoose: Choose the Next Item in a CAT

In catIrt: Simulate IRT-Based Computerized Adaptive Tests

Choose the Next Item in a CAT

Description

itChoose chooses the next item in a CAT based on the remaining items and a variety of item selection algorithms.

Usage

itChoose( left_par, mod = c("brm", "grm"), 
          numb = 1, n.select = 1,
          cat_par = NULL, cat_resp = NULL, cat_theta = NULL,
          select = c("UW-FI", "LW-FI", "PW-FI",
                     "FP-KL", "VP-KL", "FI-KL", "VI-KL",
                     "random"),
          at = c("theta", "bounds"),
          range = c(-6, 6), it.range = NULL,
          delta = NULL, bounds = NULL,
          ddist = dnorm, quad = 33, ... )

Arguments

left_par	numeric: a matrix of item parameters from which to choose the next item. The rows must index the items, and the columns must designate the item parameters (in the appropriate order, see catIrt). The first column of left_par must indicate the item numbers, as itChoose returns not just the item parameters but also the bank number corresponding to those parameters. See Details for more information.
mod	character: a character string indicating the IRT model. Current support is for the 3-parameter binary response model ("brm"), and Samejima's graded response model ("grm"). The contents of params must match the designation of mod. See catIrt or simIrt for more information.
numb	numeric: a scalar indicating the number of items to return to the user. If numb is less than n.select, then itChoose will randomly select numb items from the top n.select items according to the item selection algorithm.
n.select	numeric: an integer indicating the number of items to randomly select between at one time. For instance, if select is "UW-FI", at is "theta", numb is 3, and n.select is 8, then itChoose will randomly select 3 items out of the top 8 items that maximize Fisher information at cat_theta.
cat_par	numeric: either NULL or a matrix of item parameters that have already been administered in the CAT. cat_par only needs to be specified if letting select equal either "LW-FI", "PW-FI", "VP-KL" or "VI-KL". The format of cat_par must be the same as the format of left_par. See Details for more information.
cat_resp	numeric: either NULL or a vector of responses corresponding to the items specified in cat_par. cat_par only needs to be specified if letting select equal either "LW-FI" or "PW-FI".
cat_theta	numeric: either NULL or a scalar corresponding to the current ability estimate. cat_theta is not needed if selecting items at "bounds" or using "LW-FI" or "PW-FI" as the item selection algorithm.
select	character: a character string indicating the desired item selection method. Items can be selected either through maximum Fisher information or Kullback-Leibler divergence methods or randomly. The Fisher information methods include "UW-FI": unweighted Fisher information at a point. "LW-FI": Fisher information weighted across the likelihood function. "PW-FI": Fisher information weighted across the posterior distribution of θ. And the Kullback-Leibler divergence methods include "FP-KL": pointwise KL divergence between [P +/- delta], where P is either the current θ estimate or a classification bound. "VP-KL": pointwise KL divergence between [P +/- delta/sqrt(n)], where n is the number of items given to this point in the CAT. "FI-KL": KL divergence integrated along [P -/+ delta] with respect to P "VI-KL": KL divergence integrated along [P -/+ delta/sqrt(n)] with respect to P. See Details for more information.
at	character: a character string indicating where to select items.
range	numeric: a 2-element numeric vector indicating the range of values that itChoose should average over if select equals "LW-FI" or "PW-FI".
it.range	numeric: Either a 2-element numeric vector indicating the minimum and maximum allowed difficulty parameters for selected items (only if mod is equal to "brm") or NULL indicating no item parameter restrictions.
delta	numeric: a scalar indicating the multiplier used in item selection if a Kullback-Leibler method is chosen. For fixed-point KL divergence, delta is frequently .1 or .2, whereas in variable-point KL divergence, delta usually corresponds to 95 or 97.5 percentiles on a normal distribution.
bounds	numeric: a vector of fixed-points/bounds from which to select items if at equals "bounds".
ddist	function: a function indicating how to calculate prior densities if select equals "PW-FI" (i.e., weighting Fisher information on the posterior distribution). See catIrt for more information.
quad	numeric: a scalar indicating the number of quadrature points when select equals "LW-FI" or "PW-FI". See Details for more information.
...	arguments passed to ddist, usually distribution parameters identified by name.

Details

The function itChoose returns the next item(s) to administer in a CAT environment. The item selection algorithms fall into three major types: Fisher information, Kullback-Leibler divergence, and random.

• If choosing items based on Fisher information (select equals "UW-FI", "LW-FI", or "PW-FI"), then items are selected based on some aggregation of Fisher information (see FI). The difference between the three Fisher information methods are the weighting functions used (see van der Linden, 1998; Veerkamp & Berger, 1997). Let

  I(w_{ij} | a_j, b_j, c_j) = \int_{-∞}^{∞} w_{ij}I_j(θ)μ(dθ)

  be the "average" Fisher information, weighted by real valued function w_{ij}. Then all three Fisher information criteria can be explained solely as using different weights. Unweighted Fisher information ("UW-FI") sets w_{ij} equal to a Dirac delta function with all of its mass either on theta (if at equals "theta") or the nearest classification bound (if at equals "bounds"). Likelihood-Weighted Fisher information ("UW-FI") sets w_{ij} equal to the likelihood function given all of the previously administered items (Veerkamp & Berger, 1997). And Posterior-Weighted Fisher information ("PW-FI") sets w_{ij} equal to the likelihood function times the prior distribution specified in ddist (van der Linden, 1998). All three algorithms select items based on maximizing the respective criterion with "UW-FI" the most popular CAT item selection algorithm and equivalent to maximizing Fisher information at a point (Pashley & van der Linden, 2010).

• If choosing items based on Kullback-Leibler divergence (select equals "FP-KL", "VP-KL", "FI-KL", or "VI-KL"), then items are selected based on some aggregation of KL divergence (see KL).

  • The Pointwise KL divergence criteria (select equals "FP-KL" and "VP-KL") compares KL divergence at two points:

    KL(w_{ij} | a_j, b_j, c_j) = KL_j(P + w_{ij} || P - w_{ij})

    The difference between "FP-KL" and "VP-KL" are the weights used. Fixed Pointwise KL divergence ("FP-KL") sets w_{ij} equal to delta, and Variable Pointwise KL divergence ("VP-KL") sets w_{ij} equal to delta multiplied by 1/√{n}, where n is equal to the number of items given to this point in the CAT (see Chang & Ying, 1996).

  • The Integral KL divergence criteria (select equals "FI-KL" and "VI-KL") integrates KL divergence across a small area:

    KL(w_{ij} | a_j, b_j, c_j) = \int_{P - w_{ij}}^{P + w_{ij}} KL_j(θ || P) dθ

    As in Pointwise KL divergence, Fixed Integral KL divergence ("FI-KL") sets w_{ij} equal to delta, and Variable Integral KL divergence ("VI-KL") sets w_{ij} equal to delta multiplied by 1/√{n} (see Chang & Ying, 1996).

  All KL divergence criteria set P equal to theta (if at equals "theta") or the nearest classification bound (if at equals "bounds") and select items based on maximizing the respective criterion.

• If select is "random", then itChoose randomly picks the next item(s) out of the remaining items in the bank.

Value

itChoose returns a list of the following elements:

params	a matrix corresponding to the next item or numb items to administer in a CAT with the first column indicating the item number
info	a vector of corresponding information for the numb items of params.
type	the type of information returned in info, which is equal to the item selection algorithm.

Author(s)

Steven W. Nydick [e-mail]

References

Chang, H.-H., & Ying, Z. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213 – 229.

Pashley, P. J., & van der Linden, W. J. (2010). Item selection and ability estimation in adaptive testing. In W. J. van der Linden & C. A. W. Glas (Eds.), Elements of adaptive testing (pp. 3 – 30). New York, NY: Springer.

van der Linden, W. J. (1998). Bayesian item selection criteria for adaptive testing. Psychometrika, 63, 201 – 216.

Veerkamp, W. J. J., & Berger, M. P. F. (1997). Some new item selection criteria for adaptive testing. Journal of Educational and Behavioral Statistics, 22, 203 – 226.

See Also

catIrt, FI, KL, mleEst, simIrt

Examples

#########################
# Binary Response Model #
#########################
## Not run: 
set.seed(888)
# generating an item bank under a binary response model:
b.params <- cbind(a = runif(100, .5, 1.5), b = rnorm(100, 0, 2), c = .2)
# simulating responses using default theta:
b.mod <- simIrt(theta = 0, params = b.params, mod = "brm")

# separating the items into "administered" and "not administered":
left_par <- b.mod$params[1:95, ]
cat_par <- b.mod$params[96:100, ]
cat_resp <- b.mod$resp[ , 96:100]

# running simIrt automatically adds the item numbers to the front!

# attempting each item selection algorithm (except random):
uwfi.it <- itChoose(left_par = left_par, mod = "brm",
                    numb = 1, n.select = 1,
                    cat_theta = 0,
                    select = "UW-FI",
                    at = "theta")
lwfi.it <- itChoose(left_par = left_par, mod = "brm",
                    numb = 1, n.select = 1,
                    cat_par = cat_par, cat_resp = cat_resp,
                    select = "LW-FI")
pwfi.it <- itChoose(left_par = left_par, mod = "brm",
                    numb = 1, n.select = 1,
                    cat_par = cat_par, cat_resp = cat_resp,
                    select = "PW-FI", ddist = dnorm, mean = 0, sd = 1)

fpkl.it <- itChoose(left_par = left_par, mod = "brm",
                    numb = 1, n.select = 1,
                    cat_theta = 0,
                    select = "FP-KL",
                    at = "theta", delta = 1.96)
vpkl.it <- itChoose(left_par = left_par, mod = "brm",
                    numb = 1, n.select = 1,
                    cat_par = cat_par, cat_theta = 0,
                    select = "VP-KL",
                    at = "theta", delta = 1.96)
fikl.it <- itChoose(left_par = left_par, mod = "brm",
                    numb = 1, n.select = 1,
                    cat_theta = 0,
                    select = "FI-KL",
                    at = "theta", delta = 1.96)
vikl.it <- itChoose(left_par = left_par, mod = "brm",
                    numb = 1, n.select = 1,
                    cat_par = cat_par, cat_theta = 0,
                    select = "VI-KL",
                    at = "theta", delta = 1.96)

# which items were the most popular?
uwfi.it$params  # 61 (b close to 0)
lwfi.it$params  # 55 (b close to -2.5)
pwfi.it$params  # 16 (b close to -0.5)
fpkl.it$params  # 61 (b close to 0)
vpkl.it$params  # 61 (b close to 0)
fikl.it$params  # 16 (b close to -0.5)
vikl.it$params  # 16 (b close to -0.5)

# if we pick the top 10 items for "FI-KL":
fikl.it2 <- itChoose(left_par = left_par, mod = "brm",
                     numb = 10, n.select = 10,
                     cat_theta = 0,
                     select = "FI-KL",
                     at = "theta", delta = 1.96)

# we find that item 61 is the third best item
fikl.it2$params

# why did "LW-FI" pick an item with a strange difficulty?
cat_resp

# because cat_resp is mostly 0 ...
# --> so the likelihood is weighted toward negative numbers.

#########################
# Graded Response Model #
#########################
set.seed(999)
# generating an item bank under a graded response model:
g.params <- cbind(runif(100, .5, 1.5), rnorm(100), rnorm(100),
                                       rnorm(100), rnorm(100), rnorm(100))
# simulating responses (so that the parameters are ordered - see simIrt)
left_par <- simIrt(theta = 0, params = g.params, mod = "grm")$params

# now we can choose the best item for theta = 0 according to FI:
uwfi.it2 <- itChoose(left_par = left_par, mod = "brm",
                     numb = 1, n.select = 1,
                     cat_theta = 0,
                     select = "UW-FI",
                     at = "theta")
uwfi.it2

## End(Not run)

catIrt documentation built on May 28, 2022, 1:09 a.m.