bp_3_raw: Brennan-Prediger (BP) agreement coefficient among multiple...

In emilelatour/lagree: Calculate various interrater agreement coefficients

Brennan-Prediger (BP) agreement coefficient among multiple raters (2, 3, +)

Description

Brennan-Prediger (BP) agreement coefficient for an arbitrary number of raters (2, 3, +) when the input data represent the raw ratings reported for each subject and each rater.

Brennan-Prediger coefficient (Brennan and Prediger (1981)) and its standard error for 2 raters when input dataset is a contingency table

Brennan-Prediger coefficient (see Brennan and Prediger(1981)) and its standard error for multiple raters when input dataset is a nxq matrix representing the distribution of raters by subject and by category. The input data "ratings" is an nxq matrix showing the number of raters by subject and category. A typical entry associated with a subject and a category, represents the number of raters who classified the subject into the specified category. Exclude all subjects that are not rated by any rater.

Usage

bp_3_raw(
  data,
  ...,
  weights = "unweighted",
  categ = NULL,
  conf_lev = 0.95,
  N = Inf,
  test_value = 0,
  alternative = "two.sided"
)

bp_2_table(
  table,
  weights = "unweighted",
  conf_lev = 0.95,
  N = Inf,
  test_value = 0,
  alternative = "two.sided"
)

bp_3_dist(
  distribution,
  weights = "unweighted",
  categ = NULL,
  conf_lev = 0.95,
  N = Inf,
  test_value = 0,
  alternative = "two.sided"
)

Arguments

data	A data frame or tibble
...	Variable (column) names containing the ratings where each column represents one rater and each row one subject.
weights	is an optional parameter that is either a string variable or a matrix. The string describes one of the predefined weights and must take one of the values ("quadratic", "ordinal", "linear", "radical", "ratio", "circular", "bipolar"). If this parameter is a matrix then it must be a square matrix qxq where q is the number of possible categories where a subject can be classified. If some of the q possible categories are not used, then it is strongly advised to specify the complete list of possible categories as a vector in parameter categ. Otherwise, only the categories reported will be used.
categ	An optional parameter representing all categories available to raters during the experiment. This parameter may be useful if some categories were not used by any rater in spite of being available to the raters.
conf_lev	The confidence level associated with the agreement coefficient’s confidence interval. Default is 0.95.
N	An optional parameter representing the total number of subjects in the target subject population. Its default value is infinity, which for all practical purposes assumes the target subject population to be very large and will not require any finite-population correction when computing the standard error.
test_value	value to test the estimated AC against. Default is 0.
alternative	a character string specifying the alternative hypothesis, must be one of "two.sided" (default), "greater" or "less".
table	A q×q matrix (or contingency table) showing the distribution of subjects by rater, where q is the number of categories. This is the only argument you must specify if you want the unweighted analysis
distribution	An nxq matrix / data frame containing the distribution of raters by subject and category. Each cell (i,k) contains the number of raters who classified subject i into category k. An n x q agreement matrix representing the distribution of raters by subjects (n) and category (q) (see calc_agree_mat to convert raw data to distribution).

Value

A tbl_df with the coefficient, standard error, lower and upper confidence limits.

A tbl_df with the coefficient, standard error, lower and upper confidence limits.

A tbl_df with the coefficient, standard error, lower and upper confidence limits.

References

1. Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters. 4th ed. Gaithersburg, MD: Advanced Analytics.

Brennan, R.L., and Prediger, D. J. (1981). Coefficient Kappa: some uses, misuses, and alternatives. Educational and Psychological Measurement, 41, 687-699.

Brennan and Prediger (1981)

1. Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters. 4th ed. Gaithersburg, MD: Advanced Analytics.

Gwet, K. L. (2014). Handbook of Inter-Rater Reliability. Gaithersburg, MD: Advanced Analytics, LLC.

1. Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement Among Raters. 4th ed. Gaithersburg, MD: Advanced Analytics.

Brennan, R.L., and Prediger, D. J. (1981). Coefficient Kappa: some uses, misuses, and alternatives. Educational and Psychological Measurement, 41, 687-699.

Examples

#  5 raters classify 10 subjects into 1 of 3 rating categories
rvary2

# More than two raters
bp_3_raw(data = rvary2,
         dplyr::starts_with("rater"))

# Two raters
bp_3_raw(data = rvary2,
         rater1:rater2)

# Another example with two raters
# two radiologists who classify 85 xeromammograms into one of four categories
# (Altman p. 403)
radiologist

bp_3_raw(data = radiologist,
         radiologist_a, radiologist_b)
ratings <- matrix(c(5, 3, 0, 0,
                    3, 11, 4, 0,
                    2, 13, 3, 4,
                    1, 2, 4, 14), ncol = 4, byrow = TRUE)

bp_2_table(table = ratings)

bp_2_table(table = ratings,
           weights = "quadratic")

bp_2_table(table = ratings,
           weights = ac_weights(categ = c(1:4),
                                weight_type = "quadratic"))

my_weights <- matrix(c(1.0000000, 0.8888889, 0.5555556, 0.0000000,
                       0.8888889, 1.0000000, 0.8888889, 0.5555556,
                       0.5555556, 0.8888889, 1.0000000, 0.8888889,
                       0.0000000, 0.5555556, 0.8888889, 1.0000000),
                     ncol = 4, byrow = TRUE)

bp_2_table(table = ratings,
           weights = my_weights)
library(tidyverse)

rvary2 <- tibble::tribble(
            ~subject, ~rater1, ~rater2, ~rater3, ~rater4, ~rater5,
                  1L,      1L,      2L,      2L,      NA,      2L,
                  2L,      1L,      1L,      3L,      3L,      3L,
                  3L,      3L,      3L,      3L,      3L,      3L,
                  4L,      1L,      1L,      1L,      1L,      3L,
                  5L,      1L,      1L,      1L,      3L,      3L,
                  6L,      1L,      2L,      2L,      2L,      2L,
                  7L,      1L,      1L,      1L,      1L,      1L,
                  8L,      2L,      2L,      2L,      2L,      3L,
                  9L,      1L,      3L,      NA,      NA,      3L,
                 10L,      1L,      1L,      1L,      3L,      3L
            )

ex_dist <- calc_agree_mat(data = rvary2,
                          dplyr::starts_with("rater"),
                          subject_id = subject)

ex_dist

bp_3_dist(distribution = ex_dist)

emilelatour/lagree documentation built on Sept. 18, 2024, 5:19 p.m.