optimize_blocks: Automatic Item Pairing Method in Forced-Choice Test...

In autoFC: Automatic Toolkit for Construction, Optimization, Scoring and Simulation of Forced-Choice Tests

Automatic Item Pairing Method in Forced-Choice Test Construction

Description

Automatic construction of forced-choice tests based on the Simulated Annealing algorithm. Allows items to be:

1. Matched in either pairs, triplets, quadruplets or blocks of any size;

2. Matched based on any number of item-level characteristics (e.g. Social desirability, factor) based on any customized criteria;

3. Matched based on person-level inter-item agreement (IIA) metrics;

4. Optimally distributed at the global scale-level to ensure specific trait-pair combinations (e.g., evenly/unevenly distributing equal and/or mixed keyed pairs)

Usage

optimize_blocks(
  blocks = NULL,
  total_items = NULL,
  temp_initial = NULL,
  temp_eta = 0.01,
  temp_cooling = 0.999,
  temp_stop_ratio = 1e-06,
  item_chars,
  char_weights = NULL,
  optim_funcs = NULL,
  n_exchange = 2,
  prob_new_item = 0.25,
  use_iia = FALSE,
  response_matrix = NULL,
  iia_weights = c(BPlin = 1, BPquad = 1, AClin = 1, ACquad = 1),
  trait_col = NULL,
  key_col = NULL,
  target_dist = NULL,
  scale_fit_weight = 1,
  prevent_overlap = FALSE
)

Arguments

blocks	An n by k integer matrix, where n is the number of item blocks and k is the number of items per block. Serves as the initial starting blocks for the automatic pairing method.
total_items	Integer value. How many items do we sample from in order to build these blocks? Should be more than number of unique values in block.
temp_initial	Initial temperature value. Can be left as NULL and be computed based on the absolute value of initial energy of blocks (Recommended), and scaled by temp_eta. In general, higher temperature represents a higher probability of accepting an inferior solution.
temp_eta	A positive numeric value. The ratio of initial temperature to initial energy of blocks, if temp_initial is not designated. Default is 0.01.
temp_cooling	A positive numeric value less than 1. Determines the reduction rate of the temperature after each iteration. Default is 0.999.
temp_stop_ratio	A positive numeric value less than 1. Iteration stops when the temperature drops below temp_stop_ratio * temp_initial. Default is 10^-6.
item_chars	An m by r data frame, where m is the total number of items to sample from, whereas r is the number of item characteristics.
char_weights	A vector of length r with weights for each item characteristic in item_chars. Should provide a weight of 0 for specific characteristics not of interest, such as item IDs.
optim_funcs	A vector of customized function names for optimizing each item characteristic within each block, with length r.
n_exchange	Integer value. Determines how many blocks are exchanged in order to produce a new solution for each iteration. Should be a value larger than 1 and less than nrow(blocks). Default is 2.
prob_new_item	A value between 0 and 1. Probability of choosing the strategy of picking a new item, when not all candidate items are used to build the FC scale. Default is 0.25.
use_iia	Logical. Are IIA metrics used when performing automatic pairing? Default is FALSE.
response_matrix	A p by m numeric matrix with scores of each of the p participants for the m items. Ignored when use_iia == FALSE.
iia_weights	A vector of length 4 indicating weights given to each IIA metric: Linearly weighted AC (Gwet, 2008; 2014); Quadratic weighted AC; Linearly weighted Brennan-Prediger (BP) Index (Brennan & Prediger, 1981; Gwet, 2014); Quadratic weighted BP.
trait_col	Character string. The name of the column in item_chars indicating the trait measured by the item. Required if target_dist is used for global scale-level fit.
key_col	Character string. The name of the column in item_chars indicating the keying direction of the item (e.g., positive/negative). Required if target_dist is used.
target_dist	A data frame detailing the target distribution of trait-pair combinations for global scale-level fit. Must contain columns trait1, trait2, match_type (e.g., "equal" or "mixed"), and target (numeric count). See build_target_dist() for a helper function to generate this.
scale_fit_weight	Numeric value. The weight applied to the global scale-level fit penalty relative to the local block-level energy. Default is 1.0. Increase this value to prioritize global trait distribution over local block matching.
prevent_overlap	Logical. If target_dist is specified, should strict prevention of trait overlap (i.e., block containing items measuring the same trait) be enforced? Default is FALSE.

Value

A list containing:

• block_initial: Initial starting block

• energy_initial: Initial energy for block_initial

• block_final: Final paired block after optimization by SA

• energy_final: Final energy for block_final

Note

The essence of SA is the probablistic acceptance of solutions inferior to the current state, which avoids getting stuck in local maxima/minima. It is also recommended to try out different values of char_weights, iia_weights, temp_eta to find out the best combination of initial temperature and energy value in order to provide optimally paired blocks.

Author(s)

Mengtong Li

References

Brennan, R. L., & Prediger, D. J. (1981). Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41(3), 687-699. https://doi.org/10.1177/001316448104100307

Gwet, K. L. (2008). Computing inter rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1), 29-48. https://doi.org/10.1348/000711006X126600

Gwet, K. L. (2014). Handbook of inter-rater reliability (4th ed.): The definitive guide to measuring the extent of agreement among raters. Gaithersburg, MD: Advanced Analytics Press.

autoFC documentation built on July 14, 2026, 5:07 p.m.