collapse = TRUE,
  comment = "#>"


The irrCAC is an R package that provides several functions for calculating various chance-corrected agreement coefficients (see overview.html for ageneral overview), their weighted versions using various weights (see weighting.html for a more detailed discusssion on the weighting of agreement coefficients). In this document, I like to show you how you would implement the benchmarking approach discussed in chapter 6 of Gwet (2014). This package closely follows the general framework of inter-rater reliability assessment presented by Gwet (2014).

In a nutshell, the problem consists of qualifying the magnitude of a given agreement coeffficient as poor, good, very good, or something else. There are essentially 3 bechmarking scales that have been mentioned in the inter-rater reliability, and which are covered in this package. These are the Landis-Koch, Altman, and Fleiss bechmarking scales defined as follows:


These are data frames that become available to you as soon as you you install the irrCAC package.

Interpreting the magnitude of agreement coeeficients

Suppose that you computed Gwet's $\mbox{AC}_1$ coefficient using raw ratings from the dataset cac.raw4raters. You now want to qualify the magnitude of this coefficient using one of the benchmarking scales. Although you would normally choose one of the 3 benchmarking scales, I will use all 3 for illustration purposes. You would proceed as follows:

  ac1 <- gwet.ac1.raw(cac.raw4raters)$est
  data.frame(ac1$coeff.val, ac1$$coeff.val, ac1$$coeff.val, ac1$$coeff.val, ac1$

Each of the functions\$coeff.val, ac1\$,\$coeff.val, ac1\$, and\$coeff.val, ac1\$ produces 2 columns: the agreement strength level, and the associated cumulative membership probability (CumProb). CumProb represents the probability that the true agreement strength level is the one associated with the probability or one that is better. I recommended in Gwet (2014) to retain an agreement strength level that is associated with ComProb that exceeds 0.95. Landis-Koch Scale Since $\mbox{AC}_1=0.775$ then according to the Landoch-Koch benchmarking scale, this agreement coefficient is deemed Moderate since the associated CumProb exceeds the thereshold of 0.95. It cannot be qualified as substantial* due to its standard error of 0.14295 being high. See chapter 4 of Gwet (2014) for a more detailed discussion on this topic.


  1. Gwet, K.L. (2014, ISBN:978-0970806284). "Handbook of Inter-Rater Reliability," 4th Edition. Advanced Analytics, LLC

Try the irrCAC package in your browser

Any scripts or data that you put into this service are public.

irrCAC documentation built on Sept. 23, 2019, 5:05 p.m.