pa_3_raw: Percent agreement coefficient among multiple raters (2, 3, +)
In emilelatour/lagree: Calculate various interrater agreement coefficients

pa_3_raw

R Documentation

Percent agreement coefficient among multiple raters (2, 3, +)

Description

Percent agreement among multiple raters (2, 3, +) when the input data represent the raw ratings reported for each subject and each rater.

Percent agreement and its standard error for 2 raters when input data set is a contingency table

Percent agreement coefficient among multiple raters (2, 3, +) when the input dataset is the distribution of raters by subject and category.

Usage

pa_3_raw(
  data,
  ...,
  weights = "unweighted",
  categ = NULL,
  conf_lev = 0.95,
  N = Inf,
  test_value = 0,
  alternative = "two.sided"
)

pa_2_table(
  table,
  weights = "unweighted",
  conf_lev = 0.95,
  N = Inf,
  test_value = 0,
  alternative = "two.sided"
)

pa_3_dist(
  distribution,
  weights = "unweighted",
  categ = NULL,
  conf_lev = 0.95,
  N = Inf,
  test_value = 0,
  alternative = "two.sided"
)

Arguments

`data`	A data frame or tibble
`...`	Variable (column) names containing the ratings where each column represents one rater and each row one subject.
`weights`	is an optional parameter that is either a string variable or a matrix. The string describes one of the predefined weights and must take one of the values ("quadratic", "ordinal", "linear", "radical", "ratio", "circular", "bipolar"). If this parameter is a matrix then it must be a square matrix qxq where q is the number of possible categories where a subject can be classified. If some of the q possible categories are not used, then it is strongly advised to specify the complete list of possible categories as a vector in parameter `categ`. Otherwise, only the categories reported will be used.
`categ`	An optional parameter representing all categories available to raters during the experiment. This parameter may be useful if some categories were not used by any rater in spite of being available to the raters.
`conf_lev`	The confidence level associated with the agreement coefficient’s confidence interval. Default is 0.95.
`N`	An optional parameter representing the total number of subjects in the target subject population. Its default value is infinity, which for all practical purposes assumes the target subject population to be very large and will not require any finite-population correction when computing the standard error.
`test_value`	value to test the estimated AC against. Default is 0.
`alternative`	a character string specifying the alternative hypothesis, must be one of "two.sided" (default), "greater" or "less".
`table`	A q×q matrix (or contingency table) showing the distribution of subjects by rater, where q is the number of categories. This is the only argument you must specify if you want the unweighted analysis
`distribution`	An nxq matrix / data frame containing the distribution of raters by subject and category. Each cell (i,k) contains the number of raters who classified subject i into category k. An n x q agreement matrix representing the distribution of raters by subjects (n) and category (q) (see `calc_agree_mat` to convert raw data to distribution).

Value

A tbl_df with the coefficient, standard error, lower and upper confidence limits.

References

Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters. 4th ed. Gaithersburg, MD: Advanced Analytics.

Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters. 4th ed. Gaithersburg, MD: Advanced Analytics.

Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters. 4th ed. Gaithersburg, MD: Advanced Analytics.

Examples

#  5 raters classify 10 subjects into 1 of 3 rating categories
rvary2

# More than two raters
pa_3_raw(data = rvary2,
         dplyr::starts_with("rater"))

# Two raters
pa_3_raw(data = rvary2,
         rater1:rater2)

# Another example with two raters
# two radiologists who classify 85 xeromammograms into one of four categories
# (Altman p. 403)
radiologist

pa_3_raw(data = radiologist,
         radiologist_a, radiologist_b)

ratings <- matrix(c(5, 3, 0, 0,
                    3, 11, 4, 0,
                    2, 13, 3, 4,
                    1, 2, 4, 14), ncol = 4, byrow = TRUE)

pa_2_table(table = ratings)

pa_2_table(table = ratings,
           weights = "quadratic")

pa_2_table(table = ratings,
           weights = ac_weights(categ = c(1:4),
                                weight_type = "quadratic"))

my_weights <- matrix(c(1.0000000, 0.8888889, 0.5555556, 0.0000000,
                       0.8888889, 1.0000000, 0.8888889, 0.5555556,
                       0.5555556, 0.8888889, 1.0000000, 0.8888889,
                       0.0000000, 0.5555556, 0.8888889, 1.0000000),
                     ncol = 4, byrow = TRUE)

pa_2_table(table = ratings,
           weights = my_weights)
library(tidyverse)

rvary2 <- tibble::tribble(
            ~subject, ~rater1, ~rater2, ~rater3, ~rater4, ~rater5,
                  1L,      1L,      2L,      2L,      NA,      2L,
                  2L,      1L,      1L,      3L,      3L,      3L,
                  3L,      3L,      3L,      3L,      3L,      3L,
                  4L,      1L,      1L,      1L,      1L,      3L,
                  5L,      1L,      1L,      1L,      3L,      3L,
                  6L,      1L,      2L,      2L,      2L,      2L,
                  7L,      1L,      1L,      1L,      1L,      1L,
                  8L,      2L,      2L,      2L,      2L,      3L,
                  9L,      1L,      3L,      NA,      NA,      3L,
                 10L,      1L,      1L,      1L,      3L,      3L
            )

ex_dist <- calc_agree_mat(data = rvary2,
                          dplyr::starts_with("rater"),
                          subject_id = subject)

ex_dist

pa_3_dist(distribution = ex_dist)

emilelatour/lagree documentation built on Sept. 18, 2024, 5:19 p.m.