CohenKappa: Cohen's Kappa and Weighted Kappa In AndriSignorell/DescTools: Tools for Descriptive Statistics

Description

Computes the agreement rates Cohen's kappa and weighted kappa and their confidence intervals.

Usage

 1 2 CohenKappa(x, y = NULL, weights = c("Unweighted", "Equal-Spacing", "Fleiss-Cohen"), conf.level = NA, ...) 

Arguments

 x can either be a numeric vector or a confusion matrix. In the latter case x must be a square matrix. y NULL (default) or a vector with compatible dimensions to x. If y is provided, table(x, y, ...) is calculated. In order to get a square matrix, x and y are coerced to factors with synchronized levels. (Note, that the vector interface can not be used together with weights.) weights either one out of "Unweighted", "Equal-Spacing", "Fleiss-Cohen", which will calculate the weights accordingly, or a user-specified matrix having the same dimensions as x containing the weights for each cell. conf.level confidence level of the interval. If set to NA (which is the default) no confidence intervals will be calculated. ... further arguments are passed to the function table, allowing i.e. to set useNA. This refers only to the vector interface.

Details

Cohen's kappa is the diagonal sum of the (possibly weighted) relative frequencies, corrected for expected values and standardized by its maximum value.
The equal-spacing weights (see Cicchetti and Allison 1971) are defined by

1 - \frac{|i - j|}{r - 1}

r being the number of columns/rows, and the Fleiss-Cohen weights by

1 - \frac{(i - j)^2}{(r - 1)^2}

The latter attaches greater importance to closer disagreements.

Data can be passed to the function either as matrix or data.frame in x, or as two numeric vectors x and y. In the latter case table(x, y, ...) is calculated. Thus NAs are handled the same way as table does. Note that tables are by default calculated without NAs. The specific argument useNA can be passed via the ... argument.
The vector interface (x, y) is only supported for the calculation of unweighted kappa. For 2 factors with different levels we cannot ensure a reproducible construction of a confusion table, which is independent of the order of x and y. All weights might lead to inconsistent results. Thus the function will raise an error in such cases.

Value

if no confidence intervals are requested: the estimate as numeric value

else a named numeric vector with 3 elements

 kappa estimate lwr.ci lower confidence interval upr.ci upper confidence interval

Author(s)

David Meyer <david.meyer@r-project.org>, some slight changes Andri Signorell <andri@signorell.net>

References

Cohen, J. (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.

Everitt, B.S. (1968), Moments of statistics kappa and weighted kappa. The British Journal of Mathematical and Statistical Psychology, 21, 97-103.

Fleiss, J.L., Cohen, J., and Everitt, B.S. (1969), Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, 72, 332-327.

Cicchetti, D.V., Allison, T. (1971) A New Procedure for Assessing Reliability of Scoring EEG Sleep Recordings American Journal of EEG Technology, 11, 101-109.

CronbachAlpha, KappaM
  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 # from Bortz et. al (1990) Verteilungsfreie Methoden in der Biostatistik, Springer, pp. 459 m <- matrix(c(53, 5, 2, 11, 14, 5, 1, 6, 3), nrow=3, byrow=TRUE, dimnames = list(rater1 = c("V","N","P"), rater2 = c("V","N","P")) ) # confusion matrix interface CohenKappa(m, weight="Unweighted") # vector interface x <- Untable(m) CohenKappa(x$rater1, x$rater2, weight="Unweighted") # pairwise Kappa rating <- data.frame( rtr1 = c(4,2,2,5,2, 1,3,1,1,5, 1,1,2,1,2, 3,1,1,2,1, 5,2,2,1,1, 2,1,2,1,5), rtr2 = c(4,2,3,5,2, 1,3,1,1,5, 4,2,2,4,2, 3,1,1,2,3, 5,4,2,1,4, 2,1,2,3,5), rtr3 = c(4,2,3,5,2, 3,3,3,4,5, 4,4,2,4,4, 3,1,1,4,3, 5,4,4,4,4, 2,1,4,3,5), rtr4 = c(4,5,3,5,4, 3,3,3,4,5, 4,4,3,4,4, 3,4,1,4,5, 5,4,5,4,4, 2,1,4,3,5), rtr5 = c(4,5,3,5,4, 3,5,3,4,5, 4,4,3,4,4, 3,5,1,4,5, 5,4,5,4,4, 2,5,4,3,5), rtr6 = c(4,5,5,5,4, 3,5,4,4,5, 4,4,3,4,5, 5,5,2,4,5, 5,4,5,4,5, 4,5,4,3,5) ) PairApply(rating, FUN=CohenKappa, symmetric=TRUE) # Weighted Kappa cats <- c("<10%", "11-20%", "21-30%", "31-40%", "41-50%", ">50%") m <- matrix(c(5,8,1,2,4,2, 3,5,3,5,5,0, 1,2,6,11,2,1, 0,1,5,4,3,3, 0,0,1,2,5,2, 0,0,1,2,1,4), nrow=6, byrow=TRUE, dimnames = list(rater1 = cats, rater2 = cats) ) CohenKappa(m, weight="Equal-Spacing") # supply an explicit weight matrix ncol(m) (wm <- outer(1:ncol(m), 1:ncol(m), function(x, y) { 1 - ((abs(x-y)) / (ncol(m)-1)) } )) CohenKappa(m, weight=wm, conf.level=0.95) # however, Fleiss, Cohen and Everitt weight similarities fleiss <- matrix(c( 106, 10, 4, 22, 28, 10, 2, 12, 6 ), ncol=3, byrow=TRUE) #Fleiss weights the similarities weights <- matrix(c( 1.0000, 0.0000, 0.4444, 0.0000, 1.0000, 0.6666, 0.4444, 0.6666, 1.0000 ), ncol=3) CohenKappa(fleiss, weights)