knitr::opts_chunk$set( collapse = TRUE, fig.path = "README-", tidy = TRUE )
install.packages('caMST') # CRAN-tastic # install.packages("devtools") devtools::install_github("AnthonyRaborn/caMST") # the stable developmental version; add the argument `ref = 'devel'` for the developmental version with no stability guarantee
Here are some examples demonstrating how to use this package for various adaptive test frameworks.
source("R/mixed_adaptive_test_function.R") source("R/multistage_test_function.R") source("R/module_selection_function.R") source("R/routing_item_selection.R") source("R/iterative_maximum_likelihood_theta_estimation.R") source("R/computerized_adaptive_test_function.R")
# load the package library(caMST) # using simulated test data data(example_thetas) # 5 simulated abilities data(example_responses) # 5 simulated responses data data(example_transition_matrix)# the transition matrix for an 18 item 1-3-3 balanced design data(mst_only_items) # the CMT item bank data(mst_only_matrix) # the matrix specifying how the item data frame relates to the modules # run the CMT model using MFI for module selection resultsMFI <- multistage_test(mst_item_bank = mst_only_items, modules = mst_only_matrix, transition_matrix = example_transition_matrix, method = "BM", response_matrix = example_responses, initial_theta = 0, model = NULL, n_stages = 3, test_length = 18) resultsMFI # print a summary of the results # run the CMT model using MLWMI for module selection resultsMLWMI <- multistage_test(mst_item_bank = mst_only_items, modules = mst_only_matrix, transition_matrix = example_transition_matrix, method = "BM", response_matrix = example_responses, initial_theta = 0, model = NULL, n_stages = 3, module_select = "MLWMI", test_length = 18) resultsMFI # print a summary of the results # how good were the MFI estimates? data.frame("True Theta" = example_thetas, "Estimated Theta" = resultsMFI@final.theta.estimate, "CI95 Lower Bound" = resultsMFI@final.theta.estimate - 1.96*resultsMFI@final.theta.SEM, "CI95 Upper Bound" = resultsMFI@final.theta.estimate + 1.96*resultsMFI@final.theta.SEM) # how good were the MLWMI estimates? data.frame("True Theta" = example_thetas, "Estimated Theta" = resultsMLWMI@final.theta.estimate, "CI95 Lower Bound" = resultsMLWMI@final.theta.estimate - 1.96*resultsMLWMI@final.theta.SEM, "CI95 Upper Bound" = resultsMLWMI@final.theta.estimate + 1.96*resultsMLWMI@final.theta.SEM)
The theta estimates under CMT using the default module selection are close as all the estimates fall within the 95% confidence interval. Using the "MLWMI" module selection produces different, but similarly accurate estimates.
This is a method that follows the traditional multistage testing framework with one exception: the routing stage utilizes item-level adaptation to try to better estimate individual examinees' abilities before administering the additional stages. The current implementation allows for any three-stage design with a single routing module (e.g., 1-2-3, 1-5-3, etc.), though this will be extended for any arbitrary number of stages in future updates.
Here is an example with a 1-3-3 design that has 18 items, where each stage has the same number of items, maximum Fisher information is used for all item- and module-level adaptations, and the expected a posteriori estimator is used for the provisional and final ability estimates.
# using simulated test data data(example_thetas) # 5 simulated abilities data(example_responses) # 5 simulated responses data data(example_transition_matrix)# the transition matrix for an 18 item 1-3-3 balanced design data(cat_items) # the items designated for use in the routing module with item-level adaptation data(mst_items) # the items designated for use in the second and third modules with module-level adaptation data(example_module_items) # the matrix specifying how the item data frame relates to the modules # what does the data look like? example_thetas example_responses[,1:5] # first 5 items example_transition_matrix head(cat_items) # 564 items to choose from for the first module head(mst_items) # 18 items in the three 2nd stage modules and another 18 in the three 3rd stage modules head(example_module_items, 10) # notice that there are only 6 items in the first module! # run the Mca-MST model results <- mixed_adaptive_test(response_matrix = example_responses, cat_item_bank = cat_items, initial_theta = 0, method = "EAP", item_method = "MFI", cat_length = 6, cbControl = NULL, cbGroup = NULL, randomesque = 1, mst_item_bank = mst_items, modules = example_module_items, transition_matrix = example_transition_matrix, n_stages = 3) results # prints a summary of the results # How good was our estimate of the individual's abilities? data.frame("True Theta" = example_thetas, "Estimated Theta" = results@final.theta.estimate, "CI95 Lower Bound" = results@final.theta.estimate - 1.96*results@final.theta.SEM, "CI95 Upper Bound" = results@final.theta.estimate + 1.96*results@final.theta.SEM)
For these five individuals, the estimated ability level appears fairly close at worst and almost precisely correct at best. In all cases, the 95% confidence interval includes the simulated ability level, indicating that the Mca-MST method works well for these individuals.
In some cases, applied testing demand the use of NC scoring (even if it's not psychometrically valid). To facilitate NC tests, the multistage_test()
function accepts a list with the cutoff score for each route specified. Otherwise, the rest of the inputs are the same.
data(mst_only_items) data(example_thetas) data(example_module_items) data(example_transition_matrix) data(example_responses) # these are the same as in the previous example # however, for NC scoring, the lsit of cutoff values needs to be specified # create nc_list as explained in 'details' nc_list = list( module1 = c(4, 5, 7), module2 = c(8, 14, Inf), module3 = c(8, 14, 20), module4 = c(-Inf, 14, 20) ) # this is the ONLY difference currently! Everything else remains the same # run the example nc.results <- multistage_test( mst_item_bank = mst_only_items, modules = example_module_items, transition_matrix = example_transition_matrix, method = "BM", response_matrix = example_responses, initial_theta = 0, model = NULL, n_stages = 3, test_length = 18, nc_list = nc_list ) # printing a MST using NC scoring also shows the NC scoring method used nc.results # How well does NC scoring estimate the individual's abilities? # Using the estimation procedure from Baker for theta data.frame("True Theta" = example_thetas, "Estimated Theta" = nc.results@final.theta.estimate, "CI95 Lower Bound" = nc.results@final.theta.estimate - 1.96*results@final.theta.SEM, "CI95 Upper Bound" = nc.results@final.theta.estimate + 1.96*results@final.theta.SEM)
With this example data, the NC scoring does about as well as the previous methods in recovering the ability levels of these individuals.
Using the same individuals, we can administer a computerized adaptive test with item-level adaptation. Generally, CAT performs better than the other methods at the cost of algorithm complexity and potential difficulties maintaining item security and content balance. However, it is the golden standard for computer-based test and serves as a great test for 'best-case scenarios'.
set.seed(12345) # to keep randomesque selecting the same items # using simulated test data data(example_thetas) # 5 simulated abilities data(example_responses) # 5 simulated responsesdata data(cat_items) # using just the CAT-only routing items for the entire CAT test catResults <- computerized_adaptive_test(cat_item_bank = cat_items, response_matrix = example_responses, randomesque = 5, maxItems = 18, nextItemControl = list(criterion = "MFI", priorDist = "norm", priorPar = c(0, 1), D = 1, range = c(-4, 4), parInt = c(-4, 4, 33), infoType = "Fisher", random.seed = NULL, rule = "precision", thr = .3, nAvailable = NULL, cbControl = NULL, cbGroup = NULL)) catResults data.frame("True Theta" = example_thetas, "Estimated Theta" = catResults@final.theta.estimate, "CI95 Lower Bound" = catResults@final.theta.estimate - 1.96*catResults@final.theta.SEM, "CI95 Upper Bound" = catResults@final.theta.estimate + 1.96*catResults@final.theta.SEM)
The CAT method, using the precision rule with a value of .3 (i.e., stopping when the SEM is less than .3) and a maximum test length of 18, produces theta confidence intervals that encompass the true theta in each case. It is important to note, however, that only one individual saw less than 18 items (and he only saw 17), and this individual is the only one with an SEM of less than .3, so in reality we would probably want to increase the maxItems each individual can see in order to get a better estimate of their abilities. However, the theta estimates are accurate and all fall within the 95% CIs.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.