sa_pairing_generalized: Automatic Item Pairing Method in Forced-Choice Test...

In tspsyched/autoFC: Automatic Construction of Forced-Choice Tests

Automatic Item Pairing Method in Forced-Choice Test Construction

Description

Automatic construction of forced-choice tests based on Simulated Annealing algorithm. Allows items to be:

1. Matched in either pairs, triplets, quadruplets or blocks of any size;

2. Matched based on any number of item-level characteristics (e.g. Social desirability, factor) based on any customized criteria;

3. Matched based on person-level inter-item agreement (IIA) metrics.

Usage

sa_pairing_generalized(block, total_items, Temperature,
                              eta_Temperature = 0.01, r = 0.999,
                              end_criteria = 10^(-6),
                              item_chars, weights, FUN,
                              n_exchange = 2, prob_newitem = 0.25,
                              use_IIA = FALSE, rater_chars,
                              iia_weights = c(BPlin = 1, BPquad = 1,
                              AClin = 1, ACquad = 1))

Arguments

block	An n by k integer matrix, where n is the number of item blocks and k is the number of items per block. Serves as the initial starting blocks for the automatic pairing method.
total_items	Integer value. How many items do we sample from in order to build this block? Should be more than number of unique values in block.
Temperature	Initial temperature value. Can be left blank and be computed based on the absolute value of initial energy of block (Recommended), and scaled by eta_Temperature. In general, higher temperature represents a higher probability of accepting an inferior solution.
eta_Temperature	A positive numeric value. The ratio of initial temperature to initial energy of block, if Temperature is not designated.
r	A positive numeric value less than 1. Determines the reduction rate of Temperature after each iteration.
end_criteria	A positive numeric value less than 1. Iteration stops when temperature drops to below end_criteria * Temperature. Default to be 10^-6.
item_chars	An m by r data frame, where m is the total number of items to sample from, whether it is included in the block or not, whereas r is the number of item characteristics.
weights	A vector of length r with weights for each item characteristics in item_chars. Should provide a weight of 0 for specific characteristics not of interest, such as item ID.
FUN	A vector of customized function names for optimizing each item characteristic within each block, with length r.
n_exchange	Integer value. Determines how many blocks are exchanged in order to produce a new solution for each iteration. Should be a value larger than 1 and less than nrow(block).
prob_newitem	A value between 0 and 1. Probability of choosing the strategy of picking a new item, when not all candidate items are used to build the FC scale.
use_IIA	Logical. Are IIA metrics used when performing automatic pairing?
rater_chars	A p by m numeric matrix with scores of each of the p participants for the m items. Ignored when use_IIA == FALSE.
iia_weights	A vector of length 4 indicating weights given to each IIA metric: Linearly weighted AC (Gwet, 2008; 2014); Quadratic weighted AC; Linearly weighted Brennan-Prediger (BP) Index(Brennan & Prediger, 1981; Gwet, 2014); Quadratic weighted BP.

Value

A list containing:

block_initial Initial starting block

energy_initial Initial energy for block_initial

block_final Final paired block after optimization by SA

energy_final Final energy for block_final

Note

The essence of SA is the probablistic acceptance of solutions inferior to the current state, which avoids getting stuck in local maxima/minima. It is also recommended to try out different values of weights, iia_weights, eta_Temperature to find out the best combination of initial temperature and energy value in order to provide optimally paired blocks.

Use cal_block_energy_with_iia if inter-item agreement (IIA) metrics are needed.

Author(s)

Mengtong Li

References

Brennan, R. L., & Prediger, D. J. (1981). Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41(3), 687-699. https://doi.org/10.1177/001316448104100307

Gwet, K. L. (2008). Computing inter rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1), 29-48. https://doi.org/10.1348/000711006X126600

Gwet, K. L. (2014). Handbook of inter-rater reliability (4th ed.): The definitive guide to measuring the extent of agreement among raters. Gaithersburg, MD: Advanced Analytics Press.

Examples

## Simulate 60 items loading on different Big Five dimensions,
## with different mean and item difficulty

item_dims <- sample(c("Openness","Conscientiousness","Neuroticism",
                     "Extraversion","Agreeableness"), 60, replace = TRUE)
item_mean <- rnorm(60, 5, 2)
item_difficulty <- runif(60, -1, 1)


item_df <- data.frame(Dimensions = item_dims,
                     Mean = item_mean, Difficulty = item_difficulty)
solution <- make_random_block(60, 60, 3)


item_responses <- matrix(sample(seq(1:5), 600*60, replace = TRUE), nrow = 60, byrow = TRUE)

## Automatic pairing, without use of IIAs
## See ?facfun for information about what it does

sa_pairing_generalized(solution, 60, eta_Temperature = 0.01,
                                   r = 0.999, end_criteria = 0.001,
                                   weights = c(1,1,1),
                                   item_chars = item_df,
                                   FUN = c("facfun", "var", "var"))


## Automatic pairing, with IIAs

sa_pairing_generalized(solution, 60, eta_Temperature = 0.01,
                                   r = 0.999, end_criteria = 0.001,
                                   weights = c(1,1,1),
                                   item_chars = item_df,
                                   FUN = c("facfun", "var", "var"),
                                   use_IIA = TRUE,
                                   rater_chars = item_responses,
                                   iia_weights = c(BPlin = 1, BPquad = 1,
                                   AClin = 1, ACquad = 1))

tspsyched/autoFC documentation built on Oct. 8, 2024, 10:39 p.m.