cat_adjusted: Calculate Chance-Adjusted Agreement

In jmgirard/agreement: What the Package Does (One Line, Title Case)

Calculate Chance-Adjusted Agreement

Description

Description

Usage

cat_adjusted(
  .data,
  object = Object,
  rater = Rater,
  score = Score,
  approach = c("alpha", "gamma", "irsq", "kappa", "pi", "s"),
  categories = NULL,
  weighting = c("identity", "linear", "quadratic", "custom"),
  agreement = NULL,
  bootstrap = 2000,
  alpha_c = NULL,
  custom_weights = NULL,
  warnings = TRUE
)

Arguments

.data	Required. A matrix or data frame in tall format containing categorical data where each row corresponds to a single score (i.e., assignment of an object to a category) Cells should contain numbers or characters indicating the discrete category that the corresponding rater assigned the corresponding object to. Cells should contain NA if a particular assignment is missing (e.g., that object was not assigned to a category by that rater).
object	Optional. The name of the variable in .data identifying the object of measurement for each observation, in non-standard evaluation without quotation marks. (default = Object)
rater	Optional. The name of the variable in .data identifying the rater or source of measurement for each observation, in non-standard evaluation without quotation marks. (default = Rater)
score	Optional. The name of the variable in .data containing the categorical score or rating/code for each observation, in non-standard evaluation without quotation marks. (default = Score)
approach	Optional. A string or vector of strings specifying the chance-adjustment approach(es) to use. Currently, the "alpha", "gamma", "irsq", "kappa", "pi", and "s" approaches are available. (default = c("alpha", "gamma", "kappa", "irsq", "pi", "s"))
categories	Optional. A vector (numeric, character, or factor) containing all possible categories that objects could have been assigned to. When this argument is omitted or set to NULL, the possible categories are assumed to be those observed in .data. However, in the event that not all possible categories are observed in .data, this assumption may be misleading and so the possible categories, and their ordering, can be explicitly specified. (default = NULL)
weighting	Optional. A single string specifying the type of weighting scheme to use. Weighting schemes allow the accommodation of ordered and unordered categories with the same formulas. Currently, "identity" weights are available for unordered/nominal categories, both "linear" and "quadratic" weights are available for ordered categories, and "custom" weights can be specified via custom_weights. (default = "identity")
agreement	Optional. Either NULL or a single string specifying the formula to use in calculating percent observed agreement. Currently, "objects" is available to calculate agreement averaged across objects, "pairs" is available to calculate agreement averaged across object-rater pairs, and "kripp" is available to calculate agreement using Krippendorff's formula. NULL sets agreement to the default formula for each approach (i.e., "kripp" for Krippendorff's alpha, "pairs" for Van Oest's irsq, and "objects" for all others). (default = NULL)
bootstrap	Optional. A single non-negative integer that specifies how many bootstrap resamplings should be computed (used primarily for estimating confidence intervals and visualizing uncertainty). To skip bootstrapping, set this argument to 0. (default = 2000)
alpha_c	Optional. Either NULL or a vector of numbers corresponding to the alpha_c parameters in Van Oest's formula. If NULL, and irsq is estimated, a vector of ones will be used to implement the uniform prior coefficient. (default = NULL)
custom_weights	Optional. Either NULL or a q-by-q weight matrix where q is the number of unique categories. Weights must be between 0 (no credit) and 1 (full credit). (default = NULL)
warnings	Optional. A single logical value that specifies whether warnings should be displayed. (default = TRUE).

Value

An object of type 'cai' containing the results and details.

approach

A character vector containing the name of each approach in order

observed

A numeric vector containing the raw observed agreement according to each approach

expected

A numeric vector containing the expected chance agreement according to each approach

adjusted

A numeric vector containing the chance-adjusted agreement according to each approach. Note that these values are those typically named after each approach (e.g., this is the kappa coefficient)

boot_results

A list containing the results of the bootstrap procedure

details

A list containing the details of the analysis, such as the formatted codes, relevant counts, weighting scheme and weight matrix.

call

The function call that created these results.

References

Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Gaithersburg, MD: Advanced Analytics.

van Oest, R. (2019). A new coefficient of interrater agreement: The challenge of highly unequal category proportions. Psychological Methods, 24(4), 439-451. https://doi.org/10/ggbk3f

See Also

Other functions for categorical data: cat_specific()

jmgirard/agreement documentation built on Sept. 12, 2022, 12:39 a.m.