Description Usage Arguments Details Value Author(s) References See Also Examples

Computes the agreement rates Cohen's kappa and weighted kappa and their confidence intervals.

1 2 |

`x` |
can either be a numeric vector or a confusion matrix. In the latter case x must be a square matrix. |

`y` |
NULL (default) or a vector with compatible dimensions to |

`weights` |
either one out of |

`conf.level` |
confidence level of the interval. If set to |

`...` |
further arguments are passed to the function |

Cohen's kappa is the diagonal sum of the (possibly weighted) relative frequencies, corrected for expected values and standardized by its maximum value.

The equal-spacing weights (see Cicchetti and Allison 1971) are defined by

*1 - \frac{|i - j|}{r - 1}*

`r`

being the number of columns/rows, and the Fleiss-Cohen weights by

*1 - \frac{(i - j)^2}{(r - 1)^2}*

The latter attaches greater importance to closer disagreements.

Data can be passed to the function either as matrix or data.frame in `x`

, or as two numeric vectors `x`

and `y`

. In the latter case `table(x, y, ...)`

is calculated. Thus `NA`

s are handled the same way as `table`

does. Note that tables are by default calculated **without** NAs. The specific argument `useNA`

can be passed via the ... argument.

The vector interface `(x, y)`

is only supported for the calculation of unweighted kappa. For 2 factors with different levels we cannot ensure a reproducible construction of a confusion table, which is independent of the order of x and y. All weights might lead to inconsistent results.
Thus the function will raise an error in such cases.

if no confidence intervals are requested:
the estimate as numeric value

else a named numeric vector with 3 elements

`kappa` |
estimate |

`lwr.ci` |
lower confidence interval |

`upr.ci` |
upper confidence interval |

David Meyer <david.meyer@r-project.org>, some slight changes Andri Signorell <andri@signorell.net>

Cohen, J. (1960) A coefficient of agreement for nominal scales. *Educational and Psychological Measurement*, 20, 37-46.

Everitt, B.S. (1968), Moments of statistics kappa and weighted kappa. *The British Journal of Mathematical and Statistical Psychology*, 21, 97-103.

Fleiss, J.L., Cohen, J., and Everitt, B.S. (1969), Large sample standard errors of kappa and weighted kappa. *Psychological Bulletin*, 72, 332-327.

Cicchetti, D.V., Allison, T. (1971) A New Procedure for Assessing Reliability
of Scoring EEG Sleep Recordings *American Journal of EEG Technology*, 11,
101-109.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | ```
# from Bortz et. al (1990) Verteilungsfreie Methoden in der Biostatistik, Springer, pp. 459
m <- matrix(c(53, 5, 2,
11, 14, 5,
1, 6, 3), nrow=3, byrow=TRUE,
dimnames = list(rater1 = c("V","N","P"), rater2 = c("V","N","P")) )
# confusion matrix interface
CohenKappa(m, weight="Unweighted")
# vector interface
x <- Untable(m)
CohenKappa(x$rater1, x$rater2, weight="Unweighted")
# pairwise Kappa
rating <- data.frame(
rtr1 = c(4,2,2,5,2, 1,3,1,1,5, 1,1,2,1,2, 3,1,1,2,1, 5,2,2,1,1, 2,1,2,1,5),
rtr2 = c(4,2,3,5,2, 1,3,1,1,5, 4,2,2,4,2, 3,1,1,2,3, 5,4,2,1,4, 2,1,2,3,5),
rtr3 = c(4,2,3,5,2, 3,3,3,4,5, 4,4,2,4,4, 3,1,1,4,3, 5,4,4,4,4, 2,1,4,3,5),
rtr4 = c(4,5,3,5,4, 3,3,3,4,5, 4,4,3,4,4, 3,4,1,4,5, 5,4,5,4,4, 2,1,4,3,5),
rtr5 = c(4,5,3,5,4, 3,5,3,4,5, 4,4,3,4,4, 3,5,1,4,5, 5,4,5,4,4, 2,5,4,3,5),
rtr6 = c(4,5,5,5,4, 3,5,4,4,5, 4,4,3,4,5, 5,5,2,4,5, 5,4,5,4,5, 4,5,4,3,5)
)
PairApply(rating, FUN=CohenKappa, symmetric=TRUE)
# Weighted Kappa
cats <- c("<10%", "11-20%", "21-30%", "31-40%", "41-50%", ">50%")
m <- matrix(c(5,8,1,2,4,2, 3,5,3,5,5,0, 1,2,6,11,2,1,
0,1,5,4,3,3, 0,0,1,2,5,2, 0,0,1,2,1,4), nrow=6, byrow=TRUE,
dimnames = list(rater1 = cats, rater2 = cats) )
CohenKappa(m, weight="Equal-Spacing")
# supply an explicit weight matrix
ncol(m)
(wm <- outer(1:ncol(m), 1:ncol(m), function(x, y) {
1 - ((abs(x-y)) / (ncol(m)-1)) } ))
CohenKappa(m, weight=wm, conf.level=0.95)
# however, Fleiss, Cohen and Everitt weight similarities
fleiss <- matrix(c(
106, 10, 4,
22, 28, 10,
2, 12, 6
), ncol=3, byrow=TRUE)
#Fleiss weights the similarities
weights <- matrix(c(
1.0000, 0.0000, 0.4444,
0.0000, 1.0000, 0.6666,
0.4444, 0.6666, 1.0000
), ncol=3)
CohenKappa(fleiss, weights)
``` |

Embedding an R snippet on your website

Add the following code to your website.

For more information on customizing the embed code, read Embedding Snippets.