k_alpha | R Documentation |
This function was adapted from the paper Measuring inter-rater reliability
for nominal data – which coefficients and confidence intervals are
appropriate? by Zapf et al. Their work compared the two reliability measures
and confidence intervals to see which provide the best statistical properties
for the assessment of inter-rater reliability in different situations. With
their paper they provided supplemental R code which was adapted to this
function. Most is kept intact, but I adjusted the output to return a tibble
instead of a list and text like they had done.
Their conclusions: Fleiss' K and Krippendorff's alpha with bootstrap confidence intervals are equally suitable for the analysis of reliability of complete nominal data. The asymptotic confidence interval for Fleiss' K should not be used. In the case of missing data or data or higher than nominal order, Krippendorff’s alpha is recommended.
k_alpha(ratings_t, alpha_q = 0.05, nboot = 1000, scaling = "nominal")
ratings_t |
A matrix or data frame or tibble of the ratings. Rows =
individuals; columns = raters. Missing values coded by |
alpha_q |
Numeric; two-sided type one error, default = 0.05 |
nboot |
Integer; number of Bootstrap samples, default = 1000 |
scaling |
String; measurement scale ("nominal", "ordinal", "interval", "ratio"), default = "nominal" |
A tibble with the following columns
measure |
Name of the measure |
scale |
Measurement scale |
N_subjects |
For kappa: number of subjects without missing values; For alpha: number of subjects with two or more ratings. |
n_raters |
Number of raters |
k_categories |
Number of categories |
obs_agr |
Observed agreement for the complete cases and for all cases with at least two ratings |
estimate |
Point estimates: Fleiss' kappa and Krippendorff's alpha |
lower_asym_ci |
Lower asymptotic confidence interval |
upper_asym_ci |
Upper asymptotic confidence interval |
lower_boot_ci |
Lower bootstrap confidence interval |
upper_boot_ci |
Upper bootstrap confidence interval |
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4974794/ Efron B. Six questions raised by the bootstrap. Exploring the limits of bootstrap. Editors LePage R, Billard L. Technical Report No. 139. Division of Biostatistics, Stanford University. Wiley & Sons, New York; 1992.
ratings_t <- matrix(ncol = 3, nrow = 10,
c(5, 5, 5,
3, 5, 5,
1, 4, 4,
3, 3, 3,
4, 4, 5,
1, 3, 4,
3, 3, 3,
1, 1, 3,
2, 2, 5,
3, 3, 4),
byrow = TRUE)
k_alpha(ratings_t,
alpha_q = 0.05,
nboot = 1000,
scaling = "nominal")
k_alpha(as.data.frame(ratings_t),
alpha_q = 0.05,
nboot = 1000,
scaling = "nominal")
k_alpha(tibble::as_tibble(ratings_t),
alpha_q = 0.05,
nboot = 1000,
scaling = "nominal")
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.