In masalmon/watchme: Analysis of Autographer Images Annotations

library("knitr")
library("dplyr")

An important aspect of coding images is having a list of annotations that provides consistent results acrossed (trained) coders, or raters. Therefore one needs to check and report interrater agreement.

Simply look at disagreements

In the package there is a function for getting the times and codes of pictures for which coders disagree, which might be useful for exploring differences when e.g. training coders.

library("watchme")
data("coding1")
data("coding2")
results_list <- list(coding1, coding2)
names_list <- c('Cain', 'Abel')
watchme_output_differences(results_list = results_list,
                                names_list = names_list)

Assess interrater agreement

Nowadays the state of the art is to use Cohen's kappa for codes attributed to pictures, even if they are not independent (Aiden Doherty's personal communication). In the package we provide a function for calculating interrater agreement (or IRR) this way, which is called watchme_ira. It uses functions of the R irr package. It allows using several possibilities for defining what codes are to be compared:

one could compare the global annotations of all files, i.e. all codes at the same time. For instance compare washingYourTeeth; readingABook to washingYourTeeh; readingAMagazine for picture 1 between coder A and coder B. This is the default option.
one could look separately at each code using binary variables. For instance for washingYourTeeth comparing TRUE to TRUE and for readingABook TRUE to FALSE. For this one needs to set the by_code argument to TRUE.

In the case in which wants to compare results provided by more than two coders, another choice to be made is whether all coders are to be compared together using Fleiss Kappa, or one to one using Cohen's kappa. This is set with the one_to_one logical argument.

The comparison one wants to make depends on the context of the calculation of the IRR. When developping a new list of annotations one wants to see interrater agreement for each code, later when may want to report a single figure for the whole list of annotations.

Below are a few examples for two coders to be compared.

The default is to compare all annotations together.

data('coding1')
data('coding2')
# With two coders
results_list <- list(coding1, coding2)
names_list <- c('Cain', 'Abel')
ira_all <- watchme_ira(results_list, names_list = names_list)
kable(ira_all)

Here we compare annotations by code.

ira_codes <- watchme_ira(results_list, names_list = names_list, by_code = TRUE)
kable(ira_codes)

And then for more than two coders. If we do the comparison one by one, the resulting table has as many lines as there are possible pairs of coders. Here we compare all annotations together but even when comparing more than two coders you can do it by group of codes or by code.

results_list2 <- list(coding1, coding1, coding2)
names_list2 <- c('Riri', 'Fifi', 'Loulou')
watchme_ira(results_list2, names_list = names_list2, one_to_one = FALSE) %>%
  kable()

watchme_ira(results_list2, names_list = names_list2, one_to_one = TRUE)%>%
  kable()