Weighted Chance-corrected Agreement Coefficients"
In irrCAC: Computing Chance-Corrected Agreement Coefficients (CAC)

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

library(irrCAC)

Abstract

The irrCAC is an R package that provides several functions for calculating various chance-corrected agreement coefficients. This package closely follows the general framework of inter-rater reliability assessment presented by Gwet (2014). The document overview.html provides a broad overview of the different ways you may compute various agreement coefficients using the irrCAC package. However, it is common for researchers to want to compute the weighted versions of agreement coefficients in order to account for the ordinal nature of some ratings.

This document shows you how to specify the weights to comnpute the weighted kappa coefficient, the weighted $\mbox{AC}_2$ coefficient and many others.

The different weights

A set of weights to be applied to an agreement is generally defined in the form a matrix (i.e. a square table) whose dimension is defined par the number of categories into which the rater can classify each subject. Although the user can define her own custom weights, the inter-rater reliability literature offers predefined weights, which can be generated using functions in this package as follows:

  identity.weights(1:3)
  radical.weights(1:3)
  linear.weights(1:3)
  ordinal.weights(1:3)
  quadratic.weights(1:3)
  circular.weights(1:3)
  bipolar.weights(1:3)

You may need to read chapter 3 of Gwet (2014) for a more detailed discussion of weighted agreement coefficients. Identifty weights yield unweighted coefficients and the most commonly-used weights are the quadratic and linear weights.

Weighted Agreement Coefficients

I will show here how to compute weighted agreement coefficients separately for each of the 3 ways of organizing your input ratings that are covered in this package.

The contingency table: this dataset describes ratings from 2 raters only in such a way that each row represents a rating used by rater 1 and each column a rating used by rater 2 (see cont3x3abstractors) for an example of such a dataset).
The dataset of raw ratings: this dataset is a listing of all ratings assigned each individual subject by each rater. The row represents one subject, the column represents one rater and each value is the rating that the rater assigned to the subject (you may view the package dataset cac.raw4raters as an example).
The distribution of raters: each row of this dataset represents a subject, each column a rating used by any of the raters and each dataset value is the the number of raters who assigned the specified rating to the subject (you may see distrib.6raters as an example)

Weighting ratings from a contingency table

Suppose you want to use quadratic weights to compute various weighted coefficients using the dataset cont3x3abstractors. Using the quadratic.weights function presented above, you would proceed as follows:

  cont3x3abstractors
  q <- nrow(cont3x3abstractors)
  kappa2.table(cont3x3abstractors,weights = quadratic.weights(1:q))
  scott2.table(cont3x3abstractors,weights = quadratic.weights(1:q))
  gwet.ac1.table(cont3x3abstractors,weights = quadratic.weights(1:q))
  bp2.table(cont3x3abstractors,weights = quadratic.weights(1:q))
  krippen2.table(cont3x3abstractors,weights = quadratic.weights(1:q))
  pa2.table(cont3x3abstractors,weights = quadratic.weights(1:q))

You may use any of the weight generating functions previously discusssed to change your weights.
You may also supply your own custom set of weights to these functions. No need to stick to the weights recommended in the inter-rater reliability literature.

Weighting for a dataset of raw ratings

Suppose you want to use quadratic weights to compute various weighted coefficients using the dataset cac.raw4raters. You would proceed as follows:

  pa.coeff.raw(cac.raw4raters,weights = "quadratic")$est
  gwet.ac1.raw(cac.raw4raters,weights = "quadratic")$est
  fleiss.kappa.raw(cac.raw4raters,weights = "quadratic")$est
  krippen.alpha.raw(cac.raw4raters,weights = "quadratic")$est
  conger.kappa.raw(cac.raw4raters,weights = "quadratic")$est
  bp.coeff.raw(cac.raw4raters,weights = "quadratic")$est

If you want to use a set of weights other than quadratic weights you may replace "quadratic" with one of the following strings of characters: "ordinal", "linear", "radical", "ratio", "circular", or "bipolar".
You may also supply your own custom set of weights to these functions. In this case, the value of the weight parameter will not be a string of characters. Instead, it will be an actual matrix containing the weight values you have decided to use.

Weighting when input data is the distribution of raters by subject and rating

The calculation of weighted agreement coefficients for this type of datasets is very similar to the calculation of agreement coefficients with contingency tables. You need to use the weight-generating functions such as the quadratic.weights() fucntion to which the vector of ratings is supplied as parameter. As an example, suppose you want to compute various weighted agreement coefficients using quadratic weights and the distrib.6raters dataset. This would be accomplished as follows:

  q <- ncol(distrib.6raters)
  gwet.ac1.dist(distrib.6raters,weights = quadratic.weights(1:q))
  fleiss.kappa.dist(distrib.6raters,weights = quadratic.weights(1:q))
  krippen.alpha.dist(distrib.6raters,weights = quadratic.weights(1:q))
  bp.coeff.dist(distrib.6raters,weights = quadratic.weights(1:q))