KL: Calculate Kullback-Leibler Divergence for IRT Models

In catIrt: Simulate IRT-Based Computerized Adaptive Tests

Calculate Kullback-Leibler Divergence for IRT Models

Description

KL calculates the IRT implementation of Kullback-Leibler divergence for various IRT models given a vector of ability values, a vector/matrix of item responses, an IRT model, and a value indicating the half-width of an indifference region.

Usage

KL( params, theta, delta = .1 )
## S3 method for class 'brm'
KL( params, theta, delta = .1 )
## S3 method for class 'grm'
KL( params, theta, delta = .1 )

Arguments

params	numeric: a vector or matrix of item parameters. If specified as a matrix, the rows must index the items, and the columns must designate the item parameters. Furthermore, if calculating expected information, the number of rows must match the number of columns of resp. The class of params must match the model: either "brm" or "grm". For the binary response model, params must either be a 3-dimensional vector or a 3-column matrix. See Details for more information.
theta	numeric: a vector of ability values, one for each simulee. When performing a classification CAT, theta should be the boundary points for which to choose the next item.
delta	numeric: a scalar or vector indicating the half-width of the indifference KL will estimate the divergence between θ - δ and θ + δ using θ + δ as the "true model." If delta is a vector, then KL will use recycling to make the length of theta and delta match. See Details for more information.

Details

The function KL returns item divergence and test divergence for the binary response model ("brm") and the graded response model ("grm"). KL-divergence is defined as the following:

KL(θ_2 || θ_1) = E_{θ_2}\log[L(θ_2)/L(θ_1)]

where L(θ) stands for the likelihood of θ. Essentially, KL-divergence is the expected log-likelihood gain when using the true model in place of an alternative model.

For the binary response model, KL-divergence for an item simplifies to the following:

KL_j(θ_2 || θ_1)_j = p_j(θ_2)\log[p_j(θ_2)/p_j(θ_1)] + [1 - p_j(θ_2)]\log[(1 - p_j(θ_2))/(1 - p_j(θ_1))]

where p_{ij} is the probability of response, as indicated in the help page for simIrt

For the graded response model, KL-divergence for an item simplifies to the following:

KL_j(θ_2 || θ_1) = ∑_k{P_{jk}(θ_2)\log[P_{jk}(θ)2/P_{jk}(θ_1)]}

where P_{jk}(θ_2) is the probability of θ_2 responding in category k as indicated in the help page for simIrt. See Eggen (1999) as applied to classification CAT and van der Linden and Pashley (2010) more generally.

Because of the properties of likelihood functions in item response models, test information is simply the sum of the item informations, or:

KL(θ_2 || θ_1) = ∑_jKL_j(θ_2 || θ_1)

KL is frequently used to select items in a classification CAT where the hypotheses (e.g. being in one category versus another category are well defined). If "being in the upper category" is θ_2 and "being in the lower category" is θ_1, then θ_2 = B + δ and θ_1 = B - δ where B is the boundary separating the lower category from the upper category. Conversely, if using KL to select items in a precision CAT, then θ_2 = \hat{θ}_i + δ and θ_1 = \hat{θ}_i where \hat{θ}_i is the current, best estimate of θ. See catIrt for more information.

Value

KL, KL.brm, and KL.grm return a list of the following elements:

item	either: (1) a N \times J matrix of item information for each simulee to each item; (2) a J-length vector of item information for one simulee to each item; or (3) an N-length vector of item information for all simulees to one item, depending on the dimensions of params, theta, annd delta.
test	an N-length vector of test information, one for each simulee. Test information is the sum of item information across items. See Details for more information.

Note

Kullback-Leibler divergence in IRT is not true KL divergence, as the expectation is with respect to a model that is not necessarily true. Furthermore, it is not reciprocal, as KL(θ_1 || θ_2) \neq KL(θ_2 || θ_1). There have been other KL-based item selection measures proposed, including global information. See Eggen (1999) and itChoose.

Author(s)

Steven W. Nydick [e-mail]

References

Eggen, T. J. H. M. (1999). Item selection in adaptive testing with the sequential probability ratio test. Applied Psychological Measurement, 23, 249 – 261.

Kullback, S., & Leibler, R. A. (1951). On information and sufficiency. The Annals of Mathematical Statistics, 22, 79 – 86.

van dr Linden, W. J. & Pashley, P. J. (2010). Item selection and ability estimation in adaptive testing. In W. J. van der Linden & C. A. W. Glas (Eds.), Elements of Adaptive Testing. New York, NY: Springer.

See Also

catIrt, FI, itChoose, simIrt

Examples

#########################
# Binary Response Model #
#########################

## 1 ##
set.seed(888)
# generating random theta:
theta <- rnorm(20)
# generating an item bank under a 3-parameter binary response model:
b.params <- cbind(a = runif(100, .5, 1.5), b = rnorm(100, 0, 2), c = .2)

# you can indicate class of params or extract it from simIrt object:
class(b.params) <- "brm"

# calculating KL information with delta = .1:
k.info1 <- KL(params = b.params, theta = theta, delt = .1)
# changing delta to .2
k.info2 <- KL(params = b.params, theta = theta, delt = .2)

# notice how the overall information has increased when increasing delt:
k.info1$test; k.info2$test

# also compare with Fisher information:
f.info <- FI(params = b.params, theta = theta, type = "expected")

k.info2$test; f.info$test

# Fisher information is much higher because of how it weighs things.

## 2 ##
# we can maximize information at a boundary - say "0":
k.info3 <- KL(params = b.params, theta = 0, delta = .1)
b.params[which.max(k.info3$item), ]

# notice how the a parameter is high while the b parameter is close to
# 0, so item selection is working.

# does Fisher information choose a different item?
f.info2 <- FI(params = b.params, theta = 0, type = "expected")
b.params[which.max(f.info2$item), ]

# nope - although with more items, who knows?


#########################
# Graded Response Model #
#########################

## 1 ##
set.seed(999)
# generating random theta
theta <- rnorm(20)
# generating an item bank under a graded response model:
g.params <- cbind(runif(100, .5, 1.5), rnorm(100), rnorm(100),
                                       rnorm(100), rnorm(100), rnorm(100))
# simulating responses (so that the parameters are ordered - see simIrt)
g.params <- simIrt(theta = theta, params = g.params, mod = "grm")$params[ , -1]

# we can calculate KL information as before, noting that class(g.params) is "grm"
class(g.params)     # so we don't need to set it ourselves

# and now KL info with delt = .1
k.info4 <- KL(theta = theta, params = g.params)

# KL information is higher when more boundaries
k.info4$test
k.info1$test

# Note: k.info1 would be exactly the same if calculated with the "grm"
#       rather than the "brm"

## 2 ##
# we can also maximize information at boundary "0"
k.info5 <- KL(params = g.params, theta = 0, delta = .1)
g.params[which.max(k.info5$item), ]

# notice how the a parameter is high while the b parameters are pretty spread out.

# does Fisher information choose a different item?
f.info3 <- FI(params = g.params, theta = 0, type = "expected")
g.params[which.max(f.info3$item), ]

# nope - although with more items, who knows?

catIrt documentation built on May 28, 2022, 1:09 a.m.