knitr::opts_chunk$set( collapse = TRUE, comment = "#>" )
library(irrCAC)
The irrCAC is an R package that provides several functions for calculating various chance-corrected agreement coefficients. This package closely follows the general framework of inter-rater reliability assessment presented by Gwet (2014). The document overview.html provides a broad overview of the different ways you may compute various agreement coefficients using the irrCAC package. However, it is common for researchers to want to compute the weighted versions of agreement coefficients in order to account for the ordinal nature of some ratings.
This document shows you how to specify the weights to comnpute the weighted kappa coefficient, the weighted $\mbox{AC}_2$ coefficient and many others.
A set of weights to be applied to an agreement is generally defined in the form a matrix (i.e. a square table) whose dimension is defined par the number of categories into which the rater can classify each subject. Although the user can define her own custom weights, the inter-rater reliability literature offers predefined weights, which can be generated using functions in this package as follows:
identity.weights(1:3) radical.weights(1:3) linear.weights(1:3) ordinal.weights(1:3) quadratic.weights(1:3) circular.weights(1:3) bipolar.weights(1:3)
You may need to read chapter 3 of Gwet (2014) for a more detailed discussion of weighted agreement coefficients. Identifty weights yield unweighted coefficients and the most commonly-used weights are the quadratic and linear weights.
I will show here how to compute weighted agreement coefficients separately for each of the 3 ways of organizing your input ratings that are covered in this package.
The contingency table: this dataset describes ratings from 2 raters only in such a way that each row represents a rating used by rater 1 and each column a rating used by rater 2 (see cont3x3abstractors) for an example of such a dataset).
The dataset of raw ratings: this dataset is a listing of all ratings assigned each individual subject by each rater. The row represents one subject, the column represents one rater and each value is the rating that the rater assigned to the subject (you may view the package dataset cac.raw4raters as an example).
The distribution of raters: each row of this dataset represents a subject, each column a rating used by any of the raters and each dataset value is the the number of raters who assigned the specified rating to the subject (you may see distrib.6raters as an example)
Suppose you want to use quadratic weights to compute various weighted coefficients using the dataset cont3x3abstractors. Using the quadratic.weights function presented above, you would proceed as follows:
cont3x3abstractors q <- nrow(cont3x3abstractors) kappa2.table(cont3x3abstractors,weights = quadratic.weights(1:q)) scott2.table(cont3x3abstractors,weights = quadratic.weights(1:q)) gwet.ac1.table(cont3x3abstractors,weights = quadratic.weights(1:q)) bp2.table(cont3x3abstractors,weights = quadratic.weights(1:q)) krippen2.table(cont3x3abstractors,weights = quadratic.weights(1:q)) pa2.table(cont3x3abstractors,weights = quadratic.weights(1:q))
You may use any of the weight generating functions previously discusssed to change your weights.
You may also supply your own custom set of weights to these functions. No need to stick to the weights recommended in the inter-rater reliability literature.
Suppose you want to use quadratic weights to compute various weighted coefficients using the dataset cac.raw4raters. You would proceed as follows:
pa.coeff.raw(cac.raw4raters,weights = "quadratic")$est gwet.ac1.raw(cac.raw4raters,weights = "quadratic")$est fleiss.kappa.raw(cac.raw4raters,weights = "quadratic")$est krippen.alpha.raw(cac.raw4raters,weights = "quadratic")$est conger.kappa.raw(cac.raw4raters,weights = "quadratic")$est bp.coeff.raw(cac.raw4raters,weights = "quadratic")$est
If you want to use a set of weights other than quadratic weights you may replace "quadratic" with one of the following strings of characters: "ordinal", "linear", "radical", "ratio", "circular", or "bipolar".
You may also supply your own custom set of weights to these functions. In this case, the value of the weight parameter will not be a string of characters. Instead, it will be an actual matrix containing the weight values you have decided to use.
The calculation of weighted agreement coefficients for this type of datasets is very similar to the calculation of agreement coefficients with contingency tables. You need to use the weight-generating functions such as the quadratic.weights() fucntion to which the vector of ratings is supplied as parameter. As an example, suppose you want to compute various weighted agreement coefficients using quadratic weights and the distrib.6raters dataset. This would be accomplished as follows:
q <- ncol(distrib.6raters) gwet.ac1.dist(distrib.6raters,weights = quadratic.weights(1:q)) fleiss.kappa.dist(distrib.6raters,weights = quadratic.weights(1:q)) krippen.alpha.dist(distrib.6raters,weights = quadratic.weights(1:q)) bp.coeff.dist(distrib.6raters,weights = quadratic.weights(1:q))
Any scripts or data that you put into this service are public.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.