View source: R/classification.R
LL.CA | R Documentation |
An implementation of what has been come to be known as the "Livingston and Lewis approach" to classification consistency and accuracy, which by employing a compound beta-binomial distribution assumes that true-scores conform to the four-parameter beta distribution, and errors of measurement to the binomial distribution. Under these assumptions, the expected classification consistency and accuracy of tests can be estimated from observed outcomes and test reliability.
LL.CA(
x = NULL,
reliability,
cut,
min = 0,
max = 1,
true.model = "4P",
truecut = NULL,
output = c("accuracy", "consistency"),
failsafe = TRUE,
l = 0,
u = 1,
modelfit = c(nbins = 100, minbin = 10)
)
x |
A vector of observed scores, or a list specifying parameter values. If a list is provided, the list entries must be named after the parameters: |
reliability |
The observed-score squared correlation (i.e., proportion of shared variance) with the true-score. |
cut |
The cutoff value for classifying observations into pass or fail categories. |
min |
The minimum value possible to attain on the test. Default is 0. |
max |
The maximum value possible to attain on the test. Default is 1 (assumes that the values contained in |
true.model |
The probability distribution to be fitted to the moments of the true-score distribution. Options are |
truecut |
Optional specification of a "true" cutoff. Useful for producing ROC curves (see documentation for the |
output |
Character vector indicating which types of statistics (i.e, accuracy and/or consistency) are to be computed and included in the output. Permissible values are |
failsafe |
Logical value indicating whether to engage the automatic fail-safe defaulting to the two-parameter Beta true-score distribution if the four-parameter fitting procedure produces impermissible parameter estimates. Default is |
l |
If |
u |
If |
modelfit |
Allows for controlling the chi-square test for model fit. The argument takes either a vector of two values, or |
A list containing the estimated parameters necessary for the approach (i.e., the effective test-length and the beta distribution parameters), a chi-square test of model-fit, the confusion matrix containing estimated proportions of true/false pass/fail categorizations for a test, diagnostic performance statistics, and / or a classification consistency matrix and indices. Accuracy output includes a confusion matrix and diagnostic performance indices, and consistency output includes a consistency matrix and consistency indices p
(expected proportion of agreement between two independent test administrations), p_c
(proportion of agreement on two independent administrations expected by chance alone), and Kappa
(Cohen's Kappa).
It should be noted that this implementation differs from the original articulation of Livingston and Lewis (1995) in some respects. First, the procedure includes a number of diagnostic performance (accuracy) indices which the original procedure enables but that were not included. Second, the way consistency is calculated differs substantially from the original articulation of the procedure, which made use of a split-half approach. Rather, this implementation uses the approach to estimating classification consistency outlined by Hanson (1991).
A shiny application providing a GUI for this method is available at https://hthaa.shinyapps.io/shinybeta/ .
Livingston, Samuel A. and Lewis, Charles. (1995). Estimating the Consistency and Accuracy of Classifications Based on Test Scores. Journal of Educational Measurement, 32(2).
Hanson, Bradley A. (1991). Method of Moments Estimates for the Four-Parameter Beta Compound Binomial Model and the Calculation of Classification Consistency Indexes. American College Testing.
Lord. Frederic M. (1965). A Strong True-Score Theory, With Applications. Psychometrika, 30(3).
Lewis, Don and Burke, C. J. (1949). The Use and Misuse of the Chi-Square Test. Psychological Bulletin, 46(6).
# Generate some fictional data. Say, 1000 individuals take a test with a
# maximum score of 100 and a minimum score of 0.
set.seed(1234)
testdata <- rbinom(1000, 100, rBeta.4P(1000, 0.25, 0.75, 5, 3))
hist(testdata, xlim = c(0, 100))
# Suppose the cutoff value for attaining a pass is 50 items correct, and
# that the reliability of this test was estimated to 0.7. To estimate and
# retrieve the estimated parameters, confusion matrix, consistency and
# accuracy statistics using LL.CA():
LL.CA(x = testdata, reliability = .7, cut = 50, min = 0, max = 100)
# Suppose the true-score parameter estimation procedure arrived at
# impermissible parameter estimates (i.e., l < 0, u > 1, alpha < 0, or
# beta < 0). For example:
set.seed(9)
testdata <- rbinom(100, 25, rBeta.4P(100, 0.25, 1, 5, 3))
Beta.tp.fit(testdata, 0, 25, 25, failsafe = TRUE)
# Suppose further that you have good grounds for assuming that the lower-
# bound parameter is equal to 0.25 (e.g., the test consists of multiple-
# choice questions with four response options, leading to a 25% probability
# of guessing the correct answer per question), and good reason to believe
# that the upper-bound parameter is equal to 1 (i.e., there is no reason to
# believe that there are no members of the population who will attain a
# perfect score across all possible test-forms.) To set these lower and
# upper bounds for the fitting procedure in the LL.CA() function, set
# the argument true.model = "2p", and specify the location parameters
# l = 0.25 and u = 1:
LL.CA(testdata, 0.6287713, 12, 0, 25, true.model = "2p", l = 0.25, u = 1)
# Alternatively to supplying scores to which a true-score distribution is
# to be fit, a list with true-score distribution parameter values can be
# supplied manually along with the effective test length (see documentation
# for the ETL() function), foregoing the need for actual data. The list
# entries must be named. "l" is the lower-bound and "u" the upper-bound
# location parameters of the true-score distribution, "alpha" and "beta" for
# the shape parameters, and "etl" for the effective test-length..
trueparams <- list("l" = 0.25, "u" = 0.75, "alpha" = 5, "beta" = 3, "etl" = 50)
LL.CA(x = trueparams, cut = 50, min = 0, max = 100)
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.