In AnthonyRaborn/caMST: Mixed Computerized Adaptive Multistage Testing

knitr::opts_chunk$set(
  collapse = TRUE,
  fig.path = "README-",
  tidy = TRUE
)

caMST

Installation

install.packages('caMST') # CRAN-tastic
# install.packages("devtools")
devtools::install_github("AnthonyRaborn/caMST") # the stable developmental version; add the argument `ref = 'devel'` for the developmental version with no stability guarantee

Usage

Here are some examples demonstrating how to use this package for various adaptive test frameworks.

source("R/mixed_adaptive_test_function.R")
source("R/multistage_test_function.R")
source("R/module_selection_function.R")
source("R/routing_item_selection.R")
source("R/iterative_maximum_likelihood_theta_estimation.R")
source("R/computerized_adaptive_test_function.R")

Computer Adaptive Multistage Testing (CMT)

# load the package
library(caMST)

# using simulated test data
data(example_thetas) # 5 simulated abilities
data(example_responses) # 5 simulated responses data
data(example_transition_matrix)# the transition matrix for an 18 item 1-3-3 balanced design
data(mst_only_items) # the CMT item bank
data(mst_only_matrix) # the matrix specifying how the item data frame relates to the modules

# run the CMT model using MFI for module selection
resultsMFI <-
  multistage_test(mst_item_bank = mst_only_items, modules = mst_only_matrix, 
                  transition_matrix = example_transition_matrix,
                  method = "BM", response_matrix = example_responses, 
                  initial_theta = 0, model = NULL, n_stages = 3, test_length = 18)
resultsMFI # print a summary of the results
# run the CMT model using MLWMI for module selection
resultsMLWMI <- 
  multistage_test(mst_item_bank = mst_only_items, modules = mst_only_matrix, 
                  transition_matrix = example_transition_matrix,
                  method = "BM", response_matrix = example_responses, 
                  initial_theta = 0, model = NULL, n_stages = 3,
                  module_select = "MLWMI", test_length = 18)
resultsMFI # print a summary of the results

# how good were the MFI estimates?
data.frame("True Theta" = example_thetas,
           "Estimated Theta" = [e-mail],
           "CI95 Lower Bound" = [e-mail] -
             1.96*[e-mail],
           "CI95 Upper Bound" = [e-mail] +
             1.96*[e-mail])
# how good were the MLWMI estimates?
data.frame("True Theta" = example_thetas,
           "Estimated Theta" = [e-mail],
           "CI95 Lower Bound" = [e-mail] -
             1.96*[e-mail],
           "CI95 Upper Bound" = [e-mail] +
             1.96*[e-mail])

The theta estimates under CMT using the default module selection are close as all the estimates fall within the 95% confidence interval. Using the "MLWMI" module selection produces different, but similarly accurate estimates.

Mixed Computerized Adaptive Multistage Testing (Mca-MST)

This is a method that follows the traditional multistage testing framework with one exception: the routing stage utilizes item-level adaptation to try to better estimate individual examinees' abilities before administering the additional stages. The current implementation allows for any three-stage design with a single routing module (e.g., 1-2-3, 1-5-3, etc.), though this will be extended for any arbitrary number of stages in future updates.

Here is an example with a 1-3-3 design that has 18 items, where each stage has the same number of items, maximum Fisher information is used for all item- and module-level adaptations, and the expected a posteriori estimator is used for the provisional and final ability estimates.

# using simulated test data 
data(example_thetas) # 5 simulated abilities
data(example_responses) # 5 simulated responses data
data(example_transition_matrix)# the transition matrix for an 18 item 1-3-3 balanced design
data(cat_items) # the items designated for use in the routing module with item-level adaptation
data(mst_items) # the items designated for use in the second and third modules with module-level adaptation
data(example_module_items) # the matrix specifying how the item data frame relates to the modules

# what does the data look like?
example_thetas
example_responses[,1:5] # first 5 items
example_transition_matrix
head(cat_items) # 564 items to choose from for the first module
head(mst_items) # 18 items in the three 2nd stage modules and another 18 in the three 3rd stage modules
head(example_module_items, 10) # notice that there are only 6 items in the first module!


# run the Mca-MST model
results <- mixed_adaptive_test(response_matrix = example_responses, 
                               cat_item_bank = cat_items, initial_theta = 0,
                               method = "EAP", item_method = "MFI", 
                               cat_length = 6, cbControl = NULL, cbGroup = NULL,
                               randomesque = 1, mst_item_bank = mst_items, 
                               modules = example_module_items, 
                               transition_matrix = example_transition_matrix,
                               n_stages = 3)

results # prints a summary of the results

# How good was our estimate of the individual's abilities?
data.frame("True Theta" = example_thetas,
           "Estimated Theta" = [e-mail],
           "CI95 Lower Bound" = [e-mail] -
             1.96*[e-mail],
           "CI95 Upper Bound" = [e-mail] +
             1.96*[e-mail])

For these five individuals, the estimated ability level appears fairly close at worst and almost precisely correct at best. In all cases, the 95% confidence interval includes the simulated ability level, indicating that the Mca-MST method works well for these individuals.

Number-Correct (NC) Scoring with CMT

In some cases, applied testing demand the use of NC scoring (even if it's not psychometrically valid). To facilitate NC tests, the multistage_test() function accepts a list with the cutoff score for each route specified. Otherwise, the rest of the inputs are the same.

data(mst_only_items)
data(example_thetas)
data(example_module_items)
data(example_transition_matrix)
data(example_responses)
# these are the same as in the previous example
# however, for NC scoring, the lsit of cutoff values needs to be specified
# create nc_list as explained in 'details'
nc_list = list(
  module1 = c(4, 5, 7),
  module2 = c(8, 14, Inf),
  module3 = c(8, 14, 20),
  module4 = c(-Inf, 14, 20)
  )
# this is the ONLY difference currently! Everything else remains the same
# run the example
nc.results <- multistage_test(
  mst_item_bank = mst_only_items,
  modules = example_module_items,
  transition_matrix = example_transition_matrix,
  method = "BM",
  response_matrix = example_responses,
  initial_theta = 0,
  model = NULL,
  n_stages = 3,
  test_length = 18,
  nc_list = nc_list
  )

# printing a MST using NC scoring also shows the NC scoring method used
nc.results

# How well does NC scoring estimate the individual's abilities?
# Using the estimation procedure from Baker for theta
data.frame("True Theta" = example_thetas,
           "Estimated Theta" = [e-mail],
           "CI95 Lower Bound" = [e-mail] -
             1.96*[e-mail],
           "CI95 Upper Bound" = [e-mail] +
             1.96*[e-mail])

With this example data, the NC scoring does about as well as the previous methods in recovering the ability levels of these individuals.

Computer Adaptive Testing (CAT)

Using the same individuals, we can administer a computerized adaptive test with item-level adaptation. Generally, CAT performs better than the other methods at the cost of algorithm complexity and potential difficulties maintaining item security and content balance. However, it is the golden standard for computer-based test and serves as a great test for 'best-case scenarios'.

set.seed(12345) # to keep randomesque selecting the same items
# using simulated test data
data(example_thetas) # 5 simulated abilities
data(example_responses) # 5 simulated responsesdata
data(cat_items) # using just the CAT-only routing items for the entire CAT test

catResults <- computerized_adaptive_test(cat_item_bank = cat_items, response_matrix = example_responses, randomesque = 5, maxItems = 18, 
                                         nextItemControl = list(criterion = "MFI", priorDist = "norm", priorPar = c(0, 1), D = 1, range = c(-4, 4), parInt = c(-4, 4, 33), infoType = "Fisher", random.seed = NULL, rule = "precision", thr = .3, nAvailable = NULL, cbControl = NULL, cbGroup = NULL))

catResults

data.frame("True Theta" = example_thetas,
           "Estimated Theta" = [e-mail],
           "CI95 Lower Bound" = [e-mail] -
             1.96*[e-mail],
           "CI95 Upper Bound" = [e-mail] +
             1.96*[e-mail])

The CAT method, using the precision rule with a value of .3 (i.e., stopping when the SEM is less than .3) and a maximum test length of 18, produces theta confidence intervals that encompass the true theta in each case. It is important to note, however, that only one individual saw less than 18 items (and he only saw 17), and this individual is the only one with an SEM of less than .3, so in reality we would probably want to increase the maxItems each individual can see in order to get a better estimate of their abilities. However, the theta estimates are accurate and all fall within the 95% CIs.

AnthonyRaborn/caMST documentation built on June 27, 2022, 1:58 a.m.