krippen_3_raw: Krippendorff's alpha coefficient among multiple raters (2, 3,...

In emilelatour/lagree: Calculate various interrater agreement coefficients

Krippendorff's alpha coefficient among multiple raters (2, 3, +)

Description

Krippendorff's alpha coefficient for an arbitrary number of raters (2, 3, +) when the input data represent the raw ratings reported for each subject and each rater.

The algorithm used to compute krippendorff's alpha is very different from anything that was published on this topic. Instead, it follows the equations presented by K. Gwet (2012).

This function computes Krippendorff's alpha coefficient (see Krippendorff(1970, 1980)) and its standard error for 3 raters or more when input dataset is a nxq matrix representing the distribution of raters by subject and by category. The input data "ratings" is an nxq matrix showing the number of raters by subject and category. A typical entry associated with a subject and a category, represents the number of raters who classified the subject into the specified category. Exclude all subjects that are not rated by any rater. The algorithm used to compute krippendorff's alpha is very different from anything that was published on this topic. Instead, it follows the equations presented by K. Gwet (2010)

Usage

krippen_3_raw(
  data,
  ...,
  weights = "unweighted",
  categ = NULL,
  conf_lev = 0.95,
  N = Inf,
  test_value = 0,
  alternative = "two.sided"
)

krippen_3_dist(
  distribution,
  weights = "unweighted",
  categ = NULL,
  conf_lev = 0.95,
  N = Inf,
  test_value = 0,
  alternative = "two.sided"
)

Arguments

data	A data frame or tibble
...	Variable (column) names containing the ratings where each column represents one rater and each row one subject.
weights	is an optional parameter that is either a string variable or a matrix. The string describes one of the predefined weights and must take one of the values ("quadratic", "ordinal", "linear", "radical", "ratio", "circular", "bipolar"). If this parameter is a matrix then it must be a square matrix qxq where q is the number of possible categories where a subject can be classified. If some of the q possible categories are not used, then it is strongly advised to specify the complete list of possible categories as a vector in parameter categ. Otherwise, only the categories reported will be used.
categ	An optional parameter representing all categories available to raters during the experiment. This parameter may be useful if some categories were not used by any rater in spite of being available to the raters.
conf_lev	The confidence level associated with the agreement coefficient's confidence interval. Default is 0.95.
N	An optional parameter representing the total number of subjects in the target subject population. Its default value is infinity, which for all practical purposes assumes the target subject population to be very large and will not require any finite-population correction when computing the standard error.
test_value	value to test the estimated AC against. Default is 0.
alternative	a character string specifying the alternative hypothesis, must be one of "two.sided" (default), "greater" or "less".
distribution	An nxq matrix / data frame containing the distribution of raters by subject and category. Each cell (i,k) contains the number of raters who classified subject i into category k. An n x q agreement matrix representing the distribution of raters by subjects (n) and category (q) (see calc_agree_mat to convert raw data to distribution).

Value

A tbl_df with the coefficient, standard error, lower and upper confidence limits.

A tbl_df with the coefficient, standard error, lower and upper confidence limits.

References

Gwet, K. (2014). Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Multiple Raters, 4th Edition. Advanced Analytics, LLC;

Krippendorff (1970). "Bivariate agreement coefficients for reliability of data." Sociological Methodology,2,139-150

Krippendorff (1980). Content analysis: An introduction to its methodology (2nd ed.), New-bury Park, CA: Sage.

Gwet, K. (2012). Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Multiple Raters, 3rd Edition. Advanced Analytics, LLC; 3rd edition (March 2, 2012)

Krippendorff (1970). "Bivariate agreement coefficients for reliability of data." Sociological Methodology,2,139-150

Krippendorff (1980). Content analysis: An introduction to its methodology (2nd ed.), New-bury Park, CA: Sage.

Examples

#  5 raters classify 10 subjects into 1 of 3 rating categories
rvary2

# More than two raters
krippen_3_raw(data = rvary2,
              dplyr::starts_with("rater"))

# Two raters
krippen_3_raw(data = rvary2,
              rater1:rater2)

# Another example with two raters
# two radiologists who classify 85 xeromammograms into one of four categories
# (Altman p. 403)
radiologist

krippen_3_raw(data = radiologist,
              radiologist_a, radiologist_b)
library(tidyverse)

rvary2 <- tibble::tribble(
            ~subject, ~rater1, ~rater2, ~rater3, ~rater4, ~rater5,
                  1L,      1L,      2L,      2L,      NA,      2L,
                  2L,      1L,      1L,      3L,      3L,      3L,
                  3L,      3L,      3L,      3L,      3L,      3L,
                  4L,      1L,      1L,      1L,      1L,      3L,
                  5L,      1L,      1L,      1L,      3L,      3L,
                  6L,      1L,      2L,      2L,      2L,      2L,
                  7L,      1L,      1L,      1L,      1L,      1L,
                  8L,      2L,      2L,      2L,      2L,      3L,
                  9L,      1L,      3L,      NA,      NA,      3L,
                 10L,      1L,      1L,      1L,      3L,      3L
            )

ex_dist <- calc_agree_mat(data = rvary2,
                          dplyr::starts_with("rater"),
                          subject_id = subject)

ex_dist

krippen_3_dist(distribution = ex_dist)

emilelatour/lagree documentation built on Sept. 18, 2024, 5:19 p.m.