rcrtan

This package contains functions for criterion-referenced test and item analysis.

Test Analysis: Dependability

The functions below can be used to examine the dependability of single administration CRT assessments.

subkoviak

The subkoviak function follows Subkoviak's (1988) single administration procedures for estimating $\kappa$ and p~o~. To use the subkoviak function the data can come in one of two forms:

  1. A dataframe of dichotomously scored items
  2. A dataframe of total (raw) scores on the test. This also requires the analyst/researcher to know how many (dichotomous) items were on the test.

In both situations, the analyst/researcher needs to know the raw cut score of the test.

The procedure uses two pieces of information to estimate $\kappa$ and p~o~. They are a reliability coefficient (estimated in this function using Kuder-Richardson 20 or 21) and a standardized cut-score, which is estimated following Subkoviak (1988). The estimated $\alpha$ coefficient and standardized cut-score are used to look up the values for $\kappa$ and p~o~ in look up tables, which the function will return if look_up = TRUE

$$ z = \frac{c - 0.5 - M}{S}$$

where:

Examples: With item information

knitr::opts_chunk$set(collapse = TRUE, comment = "#>", echo=TRUE, warning=FALSE, message=FALSE, error=FALSE)
# Load the library
library(rcrtan)

# Some data
test_data <- tibble::tribble(
                 ~ID, ~Q1, ~Q2, ~Q3, ~Q4, ~Q5, ~Q6, ~Q7, ~Q8, ~Q9, ~Q10, ~Total,
               1441L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,   1L,    10L,
               1387L,  1L,  1L,  1L,  1L,  1L,  0L,  0L,  1L,  1L,   1L,     8L,
               1994L,  1L,  1L,  1L,  1L,  1L,  0L,  0L,  1L,  1L,   1L,     8L,
               1453L,  1L,  1L,  1L,  1L,  0L,  1L,  0L,  1L,  1L,   1L,     8L,
               1679L,  1L,  0L,  1L,  0L,  0L,  1L,  1L,  1L,  1L,   1L,     7L,
               1899L,  1L,  1L,  1L,  1L,  0L,  1L,  0L,  1L,  1L,   1L,     8L,
               1631L,  1L,  1L,  1L,  1L,  1L,  1L,  0L,  1L,  1L,   1L,     9L,
               1894L,  1L,  0L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,   1L,     9L,
               1206L,  0L,  0L,  1L,  1L,  0L,  1L,  0L,  1L,  0L,   1L,     5L,
               1163L,  1L,  1L,  1L,  1L,  1L,  0L,  0L,  1L,  1L,   1L,     8L,
               1964L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,   1L,    10L,
               1050L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,   1L,    10L,
               1034L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,   1L,    10L,
               1826L,  1L,  0L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,   1L,     9L,
               1973L,  1L,  1L,  1L,  1L,  0L,  1L,  0L,  1L,  1L,   1L,     8L,
               1936L,  0L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,  1L,   0L,     8L,
               1114L,  1L,  0L,  0L,  1L,  1L,  0L,  0L,  0L,  1L,   1L,     5L,
               1181L,  0L,  1L,  1L,  1L,  0L,  0L,  1L,  1L,  1L,   1L,     7L,
               1917L,  1L,  1L,  1L,  1L,  0L,  1L,  0L,  1L,  1L,   1L,     8L
               )


# Analyze the dichomous data. Use look_up = TRUE if you want to see the look up tables from Subkoviak (1988).
sub_ex_one <- subkoviak(data = test_data, items = 2:11, raw_cut_score = 8)

The output from the analysis contains five variables.

  1. z: The actual estimated standardized cut-score
  2. z_rounded: The rounded standardized cut-score used in the table look up
  3. KR_est: The estimated alpha coefficient using one the Kuder-Richardson methods
  4. agree_coef.r_0.4: The estimated agreement (p~o~) coefficient
  5. kappa_coef.r_0.4: The estimated kappa ($\kappa$) coefficient
sub_ex_one

Examples: Without item information

When item-level information is not available the items argument takes the number of dichotomous items on the test, and the total argument takes the name of the column in the dataframe with the total score.

sub_ex_two <- subkoviak(data = test_data, items = 10, raw_cut_score = 8, total = "Total")

sub_ex_two

phi_domain

The phi_domain function calculates Brown (1988) short-cut estimate of the generalizability coefficient ($\phi$ dependability) for absolute decisions.

# When item level information is known
phi_d_one <- phi_domain(data = test_data, items = 2:11)

# When only total scores and number of items on the test are known
phi_d_two <- phi_domain(data = test_data, items = 10, total = "Total")

phi_d_one

phi_d_two

phi_lambda

The phi_lambda function uses Brennan (1984) formula to estimate the dependability of a test. One difference to note between this function and subkoviak is that the cut_score argument is expressed as a proportion.

phi_l_one <- rcrtan::phi_lambda(test_data, 2:11, cut_score = 0.80)

phi_l_two <- phi_lambda(test_data, 10, cut_score = 0.80, total = 'Total')

phi_l_one

phi_l_two

References

Brennan, R. L. (1984). Estimating the dependability of the scores. In A guide to criterion-referenced test construction, (Ed.) Ronald A Berk. 231–66. Baltimore, MD: The Johns Hopkins University Press.

Brown, J. D. (1988). Short cut estimates of criterion-referenced test reliability. Language Testing, 7, 77–97.

Subkoviak, M. J. (1988). A practitioner’s guide to computation and interpretation of reliability indices for mastery tests. Journal of Educational Measurement, 25, 47–55.



Try the rcrtan package in your browser

Any scripts or data that you put into this service are public.

rcrtan documentation built on May 2, 2019, 2:10 a.m.