create_cat_design  R Documentation 
Computerized Adaptive Test (CAT) Simulation Design
Description
create_cat_design
is a helper function for
cat_sim
and cat_sim_fast
functions. It
defines the simulation design.
Ideally, there is a design element for each item. So within this design
(which is a list), there are $k$ design elements for each potentially
administered item. Each of these subdesign elements are also a list.
Usage
create_cat_design(
ip = NULL,
title = NULL,
true_ip = NULL,
first_item_rule = "fixed_theta",
first_item_par = list(theta = 0),
next_item_rule = "mfi",
next_item_par = NULL,
ability_est_rule = "eap",
ability_est_par = NULL,
final_ability_est_rule = NULL,
final_ability_est_par = NULL,
termination_rule = c("min_item", "min_se", "max_item"),
termination_par = list(min_item = 10, min_se = 0.33, max_item = 20),
testlet_rules = NULL,
exposure_control_rule = NULL,
exposure_control_par = NULL,
content_bal_rule = NULL,
content_bal_par = NULL,
ability_type = "theta"
)
Arguments
ip 
An Itempoolclass object containing item parameters,
content information, etc.
If ip = NULL this means this is an infinite item pool,
where b is on demand, c = 0 and a = 1, D = 1.7.
If true_ip argument is NULL , this item pool will
be used to generate item responses.

title 
A string value representing the title of this CAT design.

true_ip 
An Itempoolclass object which holds the true
values of item pool parameters that will be used to generate item
responses. This is an optional argument. If it is NULL
and ip is not missing, then, item responses will be
generated using ip .
Default: NULL

first_item_rule 
The method how the first item is administered.
The main effect of this is to select the first item administered
to an examinee. If, for example, first item is desired to be a
fixed one or randomly selected from the item pool, then set that
rule in next_item_rule .
Default: 'fixed_theta'
Possible values and required parameters:
 NULL
If no separate first item selection rule is
necessary, the first item will be selected using the
next_item_rule and it's parameters next_item_par .
 "fixed_theta"
Fixed starting value.
Required parameters for first_item_par argument if
this rule is selected:
 theta
The value of the initial theta estimate.
 "theta_range"
An initial theta estimate within
min_theta and max_theta will be randomly selected.
Required parameters for first_item_par argument if
this rule is selected:
 min_theta
Minimum theta value of the interval.
 max_theta
Maximum theta value of the interval.

first_item_par 
Parameters for the first item rule.
Default: list(theta = 0)

next_item_rule 
A vector of length one or length maximum test length
which is designating the next item selection rules.
Default: 'mfi'
Note that, currently, if there are testlets in an item pool and a
testlet is selected for administration using one of the methods
below, all items within that testlet will be administered regardless
of the next item selection rule.
Possible values and required parameters:
 random

Randomly select items from the item pool.
Exposure control rules and parameters will be ignored for this
selection rule.
Required parameters: None.
 mfi

Maximum Fisher Information.
Required parameters: None.
 mepv

Minimum Expected Posterior Variance.
Required Parameters:
 "var_calc_method"

Which method to use to calculate the posterior variance.
See Equation (4) of Choi and Swartz (2009), Comparison of
CAT Criteria for Polytomous Items.
Available options are:
"eap" 
Use the variance from expected a posteriori estimation.
"owen" 
Use the variance from Owen's Bayesian estimation.
For "Rasch" , "1PL" , "2PL" ,
"3PL" models this is much faster than
"eap" option above.
 b_optimal

Select item which has item difficulty that is close to the
current ability estimate.
Required parameters: None.
 fixed

Administer a fixed set of items from the item pool. This is
basically a linear fixed length test where the order of items
are predefined. Exposure control rules and parameters will be
ignored for this selection rule.
Required Parameters:
 item_id

A vector of the item IDs that should be administered.

next_item_par 
A list of length one or length maximum test length
that sets the parameters of next item selection rules. It can also
be NULL , in which case no parameters necessary for that
next item selection procedure.
Default: NULL

ability_est_rule 
A vector of length one or length maximum test length
which is designating the next item selection rules.
Default: "eap"
Possible values and required parameters:
 "eap"

Expectedaposteriori.
Required parameters:
 prior_dist

Distribution of the prior distribution.
Available values:
* norm for normal distribution,
* unif for uniform distribution.
The default value is norm .
 prior_par

A vector of prior parameters.
* For normal distribution c(0, 1) , see ?dnorm
* For uniform distribution c(3, 3) , see
?dunif
The default value is c(0, 1) .
 min_theta

Minimum possible value of theta. It is a lower bound.
The default value is 4 .
 max_theta

Maximum possible value of theta. It is an upper bound.
The default value is 4 .
 no_of_quadrature

The number of quadrature, more specifically the number of
bins the theta range should be divided. The more bins, the
more precise (and slower) the estimates will be.
The default value is 50 .
 "map"

Maximumaposteriori (Bayes Modal).
Required parameters:
 prior_dist

Distribution of the prior distribution. Currently only
available value is:
* norm for normal distribution,
The default value is norm .
 prior_par

A vector of prior parameters.
* For normal distribution c(0, 1) , see ?dnorm
* For uniform distribution c(3, 3) , see
?dunif
The default value is c(0, 1) .
 min_theta

Minimum possible value of theta. It is a lower bound.
The default value is 4 .
 max_theta

Maximum possible value of theta. It is an upper bound.
The default value is 4 .
 tol

The tolerance (precision) level of the estimate.
The default value is 0.00001 .
 "owen"

Owen's Bayesian Estimation
Required parameters:
 prior_mean
Prior mean value. The default value is
0 .
 prior_var
Prior variance value.The default value is
1 .
 "ml"

Maximum likelihood estimation using NewtonRaphson algorithm.
If this method is used, the standard error of ability estimates
are calculated using the inverse information value at this
theta estimate.
Required parameters:
 min_theta
Minimum possible value of theta. It is a
lower bound. The default value is 4.
 max_theta
Maximum possible value of theta. It is an
upper bound. The default value is 4.
 criterion
This value determines the accuracy of
estimates. Smaller values lead more accuracy but the
speed of estimation reduces as the value of criterion
decreases. The default value is 0.001.
 "eap_ml"

Expectedaposteriori until an imperfect item response string,
then switch to Maximum Likelihood estimation.
Required parameters:
 prior_dist

Distribution of the prior distribution.
Available values:
norm for normal distribution,
unif for uniform distribution.
 prior_par

A vector of prior parameters.
For normal distribution c(0, 1) , see ?dnorm
For uniform distribution c(3, 3) , see ?dunif
 min_theta

Minimum possible value of theta. It is a lower bound.
 max_theta

Maximum possible value of theta. It is an upper bound.
 no_of_quadrature

The number of quadrature, more specifically the number of
bins the theta range should be divided. The more bins, the
more precise (and slower) the estimates will be.
 "map_ml"

Maximumaposteriori until an imperfect item response string,
then switch to Maximum Likelihood estimation.
Required parameters:
 prior_dist

Distribution of the prior distribution.
Available values:
norm for normal distribution,
 prior_par

A vector of prior parameters.
For normal distribution c(0, 1) , see ?dnorm
 min_theta

Minimum possible value of theta. It is a lower bound.
 max_theta

Maximum possible value of theta. It is an upper bound.
 tol

The tolerance (precision) level of the estimate.
The default value is 0.00001 .
 "sum_score"

Simple sum score.
Required parameters: NULL

ability_est_par 
A list of length one or length maximum test length
that sets the parameters of ability estimation rules. It can also
be NULL .
* If ability_est_rule = "eap" then the default is
list(prior_dist = "norm", prior_par = list(mean = 0, sd = 2),
min_theta = 4, max_theta = 4)
* If ability_est_rule = "owen" then the default is
list(prior_mean = 0, prior_var = 1)
If it is NULL , either no parameters necessary for that
ability estimation rule or the defaults of that ability selection
rule will be selected.
If it is a list of one, it means that the parameters will be the
same throughout the test. The names of the list elements will
represent the parameter types.
A list of lists with length of maximum test length designate
different parameters for different items in the test progress.

final_ability_est_rule 
The ability estimation method that will be
used to calculate the final ability estimate. The methods and
the parameters are the same as ability_est_rule and
ability_est_par . Please see those for details.
Default: NULL

final_ability_est_par 
A list of parameters that will be used
for the method designated by the final_ability_est_rule .
Default: NULL

termination_rule 
This parameter determines how CAT algorithm decides
terminate the test.
The order of termination rules is important. The algorithm will
check the rules in that order. If for example
termination_rule = c('min_se', 'max_item') , first whether
the SE smaller than a certain value checked and if it is smaller,
then even the maximum number of items haven't been administered,
test will terminate.
The "min_item" and "max_item" has a special property
where, for "min_item" , if the number of items administered
smaller than min_item , then test will not terminate
regardless of whether other rules satisfied. Similarly, for
"max_item" , if the number of items is larger than
max_item , the test will terminate regardless of whether other
conditions satisfied or not. If both "min_item" and
"max_item" are in termination rules, then, test will end when
both conditions satisfied, i.e. when the number of items
administered is equal to or larger than max_item value in
termination_par .
The "test length" refers to "Item" objects, i.e. individual items
not testlets. For example, if an item pool has 10 testlets each
having 2 items and 15 standalone items which are not within a
testlet, then the test length can go up to 35 (2 x 10 + 15).
Default: c("min_item", "min_se", "max_item")
"termination_rule" should be a vector that composed of the
following termination rules:
"min_item" The minimum number of items should be
satisfied. If the number of administered items are equal to
or larger than this number test ends.
"max_item" The maximum number of items should not be
exceeded.
. If this is missing, then the item pool
size will be set as maximum length.
"min_se" If the standard error exceeds min_se
value, then the test will terminate.
"sprt" Sequential Probability Ratio Test (SPRT).
SPRT tests two hypotheses:
H_0: Examinee's ability \hat θ = θ_0
H_1: Examinee's ability \hat θ = θ_1
After the administration of each item, the likelihood (or
loglikelihood) of the response string is calculated at
θ_0 and θ_1. The ratio of this likelihood is
then compared to two decision points, A and B.
LR = \frac{L(θ = theta_1)}{θ = theta_0}
In order to calculate the lower (A) and upper (B)
decision points, one needs to set α and β.
α represents the rate of false positive classification
errors (0 < α < 1), i.e. examinees whose true
classification is fail but passed at the end of test. β
is the rate of false negative classification errors (0 <
β < 1), i.e. examinees whose true classification is pass but
failed at the end of test. A and B can be calculated
as:
A = \frac{1  β}{α}
B = \frac{β}{1  α}
If LR > A, examinee passes the test and if LR < B
examinee fails the test. If B < LR < A, test continues
until the maximum number of items reached (or some other test
termination criteria satisfied.)
"sprt" termination rule needs termination_par , where
the following parameters should be given in a list:
"theta_0" The highest theta value that the
test developer is willing to fail an examinee.
"theta_1" The lowest theta value that the
test developer is willing to pass an examinee.
"alpha" The rate of false positive classification
errors (0 < alpha < 1), i.e. examinees whose true
classification is fail but passed at the end of test.
"beta" The rate of false negative classification
errors (0 < beta < 1), i.e. examinees whose true
classification is pass but failed at the end of test.
Example: termination_par = list(sprt = list(theta_0 = .9,
theta_1 = .1,
alpha = 0.05,
beta = 0.05))

termination_par 
A list of termination rule parameters. This
is a named list with length equal to the length of
termination_rule argument. The names of the list elements
should correspond to the elements of termination_rule
argument.
Default: list(min_item = 10, min_se = 0.33,
max_item = 20)

testlet_rules 
A list containing arguments that specify the
rules that will be used within a testlet.
The default value is NULL where the following rules will
be applied if there is a testlet:
list(next_item_rule = "none", termination_rule = "max_item",
termination_par = list(max_item = 999)) where if a
testlet is selected all items of this testlet is selected (unless
the a testlet has more than 999 items.). Each item is selected
with the order it appears in the testlet.
It is assumed that items within testlet are administered together.
In other words, an item that does not belong to a selected testlet
cannot be administered between two items that belong to the same
testlet.
The following list elements are available:
next_item_rule The way item selection is performed
within a testlet. Following options are available:
"none" Items are selected with the order of
observed in the testlet.
"mfi" Maximum Fisher Information. The most
informative unadministered item within the testlet at
the current ability estimate is selected.
"termination_rule" The rule that should be
satisfied to stop administering items from a testlet.
Currently, only "max_item" is available. The test will
terminate when maximum number of items is reached or there
are no items left in the testlet.
"termination_par" The test termination parameters.
See the "termination_par" above in the main function for
available options.

exposure_control_rule 
A vector of length one or length maximum test
length which is designating the next item selection rules. It can
be NULL in which case there won't be any exposure control.
Default: NULL , No exposure control will be imposed
on item selection.
Possible values and required parameters:
NULL No exposure control.
 "randomesque"

Select one of the most informative num_items items.
num_items The number of items to select from.
"sympsonhetter" 
The algorithm of SympsonHetter exposure control is explained in
Sympson and Hetter (1985).
This method does not require any additional
"exposure_control_par" but each item/testlet should have
a "misc" slot like the following
misc = list(sympson_hetter_k = .75) .
When using 'sympsonhetter' exposure control rule, please ensure
that there are sufficient number of items with
'sympson_hetter_k' values 1. Otherwise, examinees might not
get a complete test and an error might be raised by the
simulation function.

exposure_control_par 
A list of length one or maximum test length
designating the exposure control for each item. If there are no
parameters it will be NULL .
Default: NULL

content_bal_rule 
Whether a content balancing is imposed on item
selection. Default value is NULL , where no content balancing
will be imposed on item selection.
Default: NULL
Possible values and required parameters:
NULL No content balancing.
 max_discrepancy
Given a target content
distribution, the content with maximum discrepancy with target
discrepancy will be administered.
Required parameters:
 target_dist
Target content ratios.
For example, suppose there are three content areas:
Geometry, Algebra and Arithmetic. If the plan for the test
is to include 30
Arithmetic items, then, the target_dist should be:
c(Geometry = .3, Arithmetic = .2, Algebra = .5). The names
in the vector should correspond to the names of the content
areas in the item pool. target_dist should include
each content area within the item pool for it to work
properly. If the sum of the target_dist is larger
than 1, it will be converted to ratios.

content_bal_par 
Parameters of content_bal_rule . A list, a
list of lists or NULL .
Default: NULL

ability_type 
The type of ability the test is measuring. By default
it is IRT based single 'theta'.
"theta" Theta for unidimensional IRT models
"multi_theta" Theta vector for multidimensional IRT
models (Not Implemented Yet).
"cdm" An attribute vector (Not Implemented Yet).
"raw_score" Raw score (i.e. total score) of an
examinee.
Default: "theta"

Value
A cat_design
object that holds the test specifications of a
CAT.
Author(s)
Emre Gonulates
References
Sympson, J., & Hetter, R. D. (1985). Controlling itemexposure rates in
computerized adaptive testing. 973–977.
See Also
cat_sim
Examples
### Example Designs ###
# Fixed length test IRT test with ability estimation EAPML
n_items < 30
ip < itempool(data.frame(a = runif(n_items, .5, 1.5), b = rnorm(n_items)))
cd < create_cat_design(ip = ip, next_item_rule = 'random',
termination_rule = 'min_item',
termination_par = list('min_item' = n_items))
cd
create_cat_design(ip = ip, next_item_rule = 'random')
n_ip < 55
ip < itempool(data.frame(a = runif(n_ip, .5, 1.5), b = rnorm(n_ip)))
# Check the default:
create_cat_design()
create_cat_design(ip = ip)
### Termination Rule ###
create_cat_design(
termination_rule = c('min_item', 'min_se', 'max_item'),
termination_par = list(min_item = 10, min_se = .33, max_item = 20))
cd < create_cat_design(ip = ip, termination_rule = c('min_item', 'min_se'),
termination_par = list(min_item = 10, min_se = .33))
### Next Item Rule ###
create_cat_design(ip = ip, next_item_rule = 'random', next_item_par = NULL)
create_cat_design(
ip = ip, termination_rule = c('min_item', 'max_item'),
termination_par = list(min_item = 20, max_item = 20),
next_item_rule = 'fixed',
next_item_par = list(item_id = ip$item_id[1:20]))
# Linear test where all of the items in the item pool administered in the
# same order as item pool
ip < generate_ip(n = 15)
create_cat_design(
ip = ip, termination_rule = c('max_item'),
termination_par = list(max_item = 15),
next_item_rule = 'fixed')
# Generate an item pool with two testlets and three standalone items and
# administer first seven items as a linear test.
ip < c(generate_testlet(n = 2, testlet_id = "t1"), generate_ip(n = 3),
generate_testlet(n = 5, testlet_id = "t2"))
create_cat_design(
ip = ip, termination_rule = c('max_item'),
termination_par = list(max_item = 7),
next_item_rule = 'fixed')
# A linear test where the item order is predefined.
ip1 < itempool(data.frame(b = rnorm(5)), item_id = paste0("i",1:5))
cd < create_cat_design(
ip = ip1,
next_item_rule = 'fixed',
next_item_par = list(item_id = c("i3", "i2", "i4", "i5", "i1")),
ability_est_rule = "eap",
termination_rule = 'max_item', termination_par = list(max_item = 5))
### Ability Estimation Rule ###
create_cat_design(
ability_est_rule = 'eap',
ability_est_par = list(prior_dist = 'unif',
prior_par = list(min = 2, max = 2),
min_theta = 4, max_theta = 4,
no_of_quadrature = 31))
create_cat_design(
ability_est_rule = 'ml',
ability_est_par = list(min_theta = 4, max_theta = 4, criterion = 0.01))
### Exposure Control ###
create_cat_design(exposure_control_rule = 'randomesque',
exposure_control_par = list(num_items = 1))
# 54321 exposure control
create_cat_design(
exposure_control_rule = 'randomesque',
exposure_control_par = lapply(c(5:1, rep(1, 15)),
function(x) list(num_items = x)))
### Content Balancing ###
create_cat_design(
content_bal_rule = 'max_discrepancy',
content_bal_par = list(target_dist = c(
Geometry = .3, `Rational Numbers` = .2, Algebra = .5)))