In LAAC-LSCP/ChildRecordsR: ChildRecordsR Reliability Analysis

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.width=7,fig.height = 8
)

The ChildRecordsR package is an R package dedicated to the analysis of annotations of daylong recordings in ChildRecordsData format. The objective of our package is to provide data aggregation functions and to analyze the reliability of annotations and annotators.

Install the Vandam data

Vandam daylong recording is a public data set, a childrecordsData format and install procedure is available at (https://github.com/LAAC-LSCP/vandam-daylong-demo)

Create a ChildRecordings class

Here you will create a class by specifying the root folder of your corpus, which needs to be formatted using ChildRecordingData specifications. By using a class, we standardize all the references to the information in your corpus. Additionally, we provide basic checks such as missing files or unreferenced files in the meta data. Try to add, misplace or erase some files to see how these checks work.

library(ChildRecordsR)
ChildRecordingsPath = "/mnt/94707AA4707A8CAC/CNRS/corpus/vandam-daylong-demo" # Use your  own path to Vandam data 
CR = ChildRecordings(ChildRecordingsPath)

All functions are based on the class (i.e., CR in our example above) to avoid problems of reference, since the class is always set up in the same way.

For the purpose of the tutorial we will need more annotation than provided initially. The next function will help us to add some dummy annotations with distortion in the rating. The first dummy annotation will have slightly modified and will be very close to the original; the second and third on will have important modifiable and could be considered as a bad annotator.

New.annotations(row.meta = CR$all.meta[2,], time.sd = 10, change.cat.prob = 0.001, new.name = "vtc_mod", CR)
New.annotations(row.meta = CR$all.meta[2,], time.sd = 1500, change.cat.prob = 0.10, new.name = "vtc_BAD", CR)
New.annotations(row.meta = CR$all.meta[2,], time.sd = 1800, change.cat.prob = 0.15, new.name = "vtc_VeryBAD", CR)

Finding annotations: Search function

Before it can provide any statistical reliability, the current package will need to find annotation segments that have been annotated by at least two annotators. The annotators could be humans or algorithms -- the package does not know the difference, so you need to think about implications. This search is performed by the find.rating.segment function, which returns a data frame with the wav filenames, the annotators' codenames, the annotation filenames and the onset and offset of the annotated segment(s) with respect to the wav.

At a minimum, you need to provide to the search function the class (i.e., CR in our example above) and the relative path to one or several wav files. The function will then find every segment annotated by any annotators in the wav files. In the following example, we provide the path to a single wav file (to speed things up, see section Analyze a corpus in the ChildRecordsR vignette for an example with multiple files):

record.file <- CR$all.meta$recording_filename[1]

find.rating.segment(CR,record.file)

Alternatively, if a specific time window is provided, the search function will find all the annotations that overlap with the time window provided.

find.rating.segment(CR,record.file,range_from = 27180000, range_to = 27240000)
find.rating.segment(CR,record.file,range_from = 27000000, range_to = 50464000)
find.rating.segment(CR,record.file,range_from = 0, range_to = 27260000)

It is also possible to find annotations for a specific set of annotators, by providing the list of their codenames:

raters <- c("its","vtc","vtc_mod","vtc_BAD")
find.rating.segment(CR,record.file,raters)

Time window and limited annotator can also be specified jointly:

search1 <- find.rating.segment(CR,record.file, raters, range_from = 12616000, range_to = 12616000*3)
search1

Measuring reliability and classification agreement

Once you have obtained information on a set of annotations with the search function and stored it in an object (search1 above), you can use the aggregate function. This function will create a table with the annotation information, and additionally convert your data into a long format, where annotations are split up into temporal bins. For instance, imagine that an annotator said FEM spoke between .5s and 1s of the wav file. This would be one row in the table format. To convert this to long format, a bin size (in milliseconds) can be provided using the cut argument. For instance, if you specify cut = 10, then that 500 millisecond vocalization by FEM becomes 50 rows, each representing 10 milliseconds of speech. By default cut is set to 100 milliseconds. The function will return a raterData class with the original format and a long format for every annotator.

rating1  = aggregate.rating(search1, CR, cut = 100)

Next, an analysis function can be called. We provide several. They all assume you have at least two annotators. It could be two automated algorithms, two humans, or one automated and one human -- this package does not treat these cases differently, so you should interpret results carefully.

The reliability or get.reliability function provides alpha, kappa and AC1 for the whole pool of annotators in your search (so this works even if you have 10 annotators). Reliability will be computed for every speaker category and a composite of all of them.

rez1 = get.reliability(rating1)

Another possible way to investigate annotators is through classification indicators, such as precision, recall, and F-score. These can only be calculated for cases with exactly two annotators. The first annotator will be assumed as the "gold" annotator, so if you have a human and an automated annotator, or someone with more versus less experience, make sure you order them so that the more reliable annotator (more likely to be "gold") is provided first.

ratercomp <- c("vtc","its")
get.classification(rating1,ratercomp)

Compare annotators

Finally, the raterComparaison function allows you to compare the impact of annotators on reliability indicators. This is useful if you have multiple human annotators and you want to check whether one of them is doing something different, or if you want to check whether your automated annotator stands out in your pool of annotators. You can only use this function when you have more than 2 annotators. The impact of a given annotator is calculated by removing that annotator from the pool of annotators, and observing the increase or decrease in the overall reliability. If the reliability indicators increase after the deletion of a specific annotator, then that annotator should be considered as having a negative impact on the annotations. In our next example, vtc_BAD has a negative impact on reliability.

comparaison = compare.rating(rating1)
plot(comparaison)

LAAC-LSCP/ChildRecordsR documentation built on July 26, 2021, 3:25 p.m.