k_alpha: Calculate Fleiss' kappa and Krippendorff's alpha

In emilelatour/lamisc: Latour's Little Helpers

Calculate Fleiss' kappa and Krippendorff's alpha

Description

This function was adapted from the paper Measuring inter-rater reliability for nominal data – which coefficients and confidence intervals are appropriate? by Zapf et al. Their work compared the two reliability measures and confidence intervals to see which provide the best statistical properties for the assessment of inter-rater reliability in different situations. With their paper they provided supplemental R code which was adapted to this function. Most is kept intact, but I adjusted the output to return a tibble instead of a list and text like they had done.

Their conclusions: Fleiss' K and Krippendorff's alpha with bootstrap confidence intervals are equally suitable for the analysis of reliability of complete nominal data. The asymptotic confidence interval for Fleiss' K should not be used. In the case of missing data or data or higher than nominal order, Krippendorff’s alpha is recommended.

Usage

k_alpha(ratings_t, alpha_q = 0.05, nboot = 1000, scaling = "nominal")

Arguments

ratings_t	A matrix or data frame or tibble of the ratings. Rows = individuals; columns = raters. Missing values coded by NA.
alpha_q	Numeric; two-sided type one error, default = 0.05
nboot	Integer; number of Bootstrap samples, default = 1000
scaling	String; measurement scale ("nominal", "ordinal", "interval", "ratio"), default = "nominal"

Value

A tibble with the following columns

measure	Name of the measure
scale	Measurement scale
N_subjects	For kappa: number of subjects without missing values; For alpha: number of subjects with two or more ratings.
n_raters	Number of raters
k_categories	Number of categories
obs_agr	Observed agreement for the complete cases and for all cases with at least two ratings
estimate	Point estimates: Fleiss' kappa and Krippendorff's alpha
lower_asym_ci	Lower asymptotic confidence interval
upper_asym_ci	Upper asymptotic confidence interval
lower_boot_ci	Lower bootstrap confidence interval
upper_boot_ci	Upper bootstrap confidence interval

References

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4974794/ Efron B. Six questions raised by the bootstrap. Exploring the limits of bootstrap. Editors LePage R, Billard L. Technical Report No. 139. Division of Biostatistics, Stanford University. Wiley & Sons, New York; 1992.

Examples

ratings_t <- matrix(ncol = 3, nrow = 10,
                    c(5, 5, 5,
                      3, 5, 5,
                      1, 4, 4,
                      3, 3, 3,
                      4, 4, 5,
                      1, 3, 4,
                      3, 3, 3,
                      1, 1, 3,
                      2, 2, 5,
                      3, 3, 4),
                    byrow = TRUE)
k_alpha(ratings_t,
        alpha_q = 0.05,
        nboot = 1000,
        scaling = "nominal")
k_alpha(as.data.frame(ratings_t),
        alpha_q = 0.05,
        nboot = 1000,
        scaling = "nominal")
k_alpha(tibble::as_tibble(ratings_t),
        alpha_q = 0.05,
        nboot = 1000,
        scaling = "nominal")

emilelatour/lamisc documentation built on April 9, 2024, 10:33 a.m.