krippen.alpha.raw: Krippendorff's alpha coefficient for an arbitrary number of...

In irrCAC: Computing Chance-Corrected Agreement Coefficients (CAC)

Description

Krippendorff's alpha coefficient for an arbitrary number of raters (2, 3, +) when the input data represent the raw ratings reported for each subject and each rater.

Usage

krippen.alpha.raw(ratings, weights = "unweighted", categ.labels = NULL,
  conflev = 0.95, N = Inf)

Arguments

ratings	An nxr matrix / data frame of ratings where each column represents one rater and each row one subject.
weights	is a mandatory parameter that is either a string variable or a matrix. The string describes one of the predefined weights and must take one of the values ("quadratic", "ordinal", "linear", "radical", "ratio", "circular", "bipolar"). If this parameter is a matrix then it must be a square matri qxq where q is the number of posssible categories where a subject can be classified. If some of the q possible categories are not used, then it is strobgly advised to specify the complete list of possible categories as a vector in parametr categ.labels. Otherwise, the program may not work.
categ.labels	An optional vector parameter containing the list of all possible ratings. It may be useful in case some of the possibe ratings are not used by any rater, they will still be used when calculating agreement coefficients. The default value is NULL. In this case, only categories reported by the raters are used in the calculations.
conflev	An optional parameter representing the confidence level associated with the confidence interval. Its default value is 0.95.
N	An optional parameter representing the population size (if any). It may be use to perform the final population correction to the variance. Its default value is infinity.

Value

A data list containing 3 objects: (1) a one-row data frame containing various statistics including the requested agreement coefficient-in this case, Krippendorff's alpha, (2) the weight matrix used in the calculations if any, and (3) the vector of categories used in the analysis. These could be categories reported by the raters, or those that were available to the raters whether they used them or not. The output data frame contains the following variables: "coeff.name" (coefficient name), "pa" (the percent agreement), "pe" (the percent chance agreement), coeff.val (Krippendorff's alpha estimate), "coeff.se (standard error), conf.int" (Krippendorff alpha's confidence interval),"p.value" (Krippendorff alpha's p-value), "w.name" (the weights' identification).

References

Gwet, K. (2014). Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Multiple Raters, 4th Edition. Advanced Analytics, LLC.

Krippendorff (1970). “Bivariate agreement coefficients for reliability of data." Sociological Methodology,2,139-150.

Krippendorff (1980). Content analysis: An introduction to its methodology (2nd ed.), New-bury Park, CA: Sage.

Examples

#The dataset "cac.raw4raters" comes with this package. Analyze it as follows:
cac.raw4raters
krippen.alpha.raw(cac.raw4raters) #Alpha coeff. , precision measures, weights & categories
krippen.alpha.raw(cac.raw4raters)$est #Krippendorff's alpha with precision measures
alpha <- krippen.alpha.raw(cac.raw4raters)$est$coeff.val #Krippendorff's alpha alone.
alpha
krippen.alpha.raw(cac.raw4raters, weights = "quadratic") #weighted alpha/ quadratic wts

Example output

   Rater1 Rater2 Rater3 Rater4
1       1      1     NA      1
2       2      2      3      2
3       3      3      3      3
4       3      3      3      3
5       2      2      2      2
6       1      2      3      4
7       4      4      4      4
8       1      1      2      1
9       2      2      2      2
10     NA      5      5      5
11     NA     NA      1      1
12     NA     NA      3     NA
$est
            coeff.name    pa   pe coeff.val coeff.se  conf.int      p.value
1 Krippendorff's Alpha 0.805 0.24   0.74342  0.14557 (0.419,1) 0.0004594257
      w.name
1 unweighted

$weights
     [,1] [,2] [,3] [,4] [,5]
[1,]    1    0    0    0    0
[2,]    0    1    0    0    0
[3,]    0    0    1    0    0
[4,]    0    0    0    1    0
[5,]    0    0    0    0    1

$categories
[1] 1 2 3 4 5

            coeff.name    pa   pe coeff.val coeff.se  conf.int      p.value
1 Krippendorff's Alpha 0.805 0.24   0.74342  0.14557 (0.419,1) 0.0004594257
      w.name
1 unweighted
[1] 0.74342
$est
            coeff.name        pa    pe coeff.val coeff.se  conf.int
1 Krippendorff's Alpha 0.9735938 0.825   0.84911  0.12913 (0.561,1)
       p.value    w.name
1 6.267448e-05 quadratic

$weights
       [,1]   [,2]   [,3]   [,4]   [,5]
[1,] 1.0000 0.9375 0.7500 0.4375 0.0000
[2,] 0.9375 1.0000 0.9375 0.7500 0.4375
[3,] 0.7500 0.9375 1.0000 0.9375 0.7500
[4,] 0.4375 0.7500 0.9375 1.0000 0.9375
[5,] 0.0000 0.4375 0.7500 0.9375 1.0000

$categories
[1] 1 2 3 4 5

irrCAC documentation built on Sept. 23, 2019, 5:05 p.m.