agreement: Agreement

In iriseekhout/Agree: Agreement and reliability between multiple raters

Agreement

Description

The default function returns the proportion of overall agreement between two or more raters. Specific agreements can be obtained as well. Specific agreement (averages over discordant cells to correct for random rater combinations). When there are 2 categories, this is equal to the positive/negative agreement. When there are more than two categories, one can either look at the agreement for one category versus the others, for example very satisfied versus rest or for one category versus one specific other category, for example very satisfies versus not satisfied.

Usage

agreement(
  data,
  specific = NULL,
  confint = FALSE,
  alpha = 0.05,
  n = nrow(data),
  k = ncol(data),
  b = 1000,
  ...
)

Arguments

data	A data frame or table with equal number of columns and rows. Or a data frame that contains the scores for each rater in each column.
specific	A character vector indicating the category for which specific agreement should be calculated. If length(specific) == 1, the named factor level is compared to all others, if length(specific) == 2 the specific agreement is of the first category level is compared to the second category level. If specific = "positive" or specific = "negative" and scores are dichotomous, the positive or negative agreement will be returned.
confint	logical vector if confidence interval for agreements are computed.
alpha	the confidence interval level
n	sample size; default n = nrow(data)
k	number of raters; default k = ncol(data)
b	number of bootstrap iterations for bootstrapped CI
...	options for sumtable if data = data.frame

Details

De confidence intervals are obtained and adjusted with a Fleis continuity correction. For the specific agreements with polytomous data, the confidence intervals are bootstrapped.

Value

An S3 object containing the proportion of overall agreement.

Examples

#dichotomous
df <- data.frame(r1=factor(c(1,0,1,0,0,1,1,0,0,0,1,1,0,1,1)),
                 r2=factor(c(1,1,1,1,0,1,1,0,0,0,1,1,0,1,0)),
                 r3=factor(c(1,1,1,0,0,0,1,1,1,0,0,1,0,1,1)),
                 r4=factor(c(1,1,1,1,1,1,1,1,1,1,1,1,1,1,1)))
table <- sumtable(df=df, ratings=c("r1", "r2", "r3", "r4"),
levels=c("0","1"))
agreement(table)
agreement(table, specific = "1")
agreement(table, specific = "1", confint = TRUE)
agreement(table, specific = "positive")
agreement(df)
agreement(df, specific = "1")
agreement(df, specific = "positive")
agreement(df, specific = "negative")
#polytomous
df <- data.frame(r1=factor(c(1,2,2,0,3,3,1,0,3,0,2,2,0,3,1)),
                 r2=factor(c(1,1,1,0,3,3,1,0,1,0,2,2,0,2,1)),
                 r3=factor(c(1,1,1,3,3,2,1,0,1,0,2,2,0,3,1)),
                 r4=factor(c(1,2,1,0,3,3,1,0,3,0,2,2,0,2,1)))
table <- sumtable(df=df, ratings=c("r1", "r2", "r3", "r4"),
levels=c("0","1", "2", "3"))
agreement(table, specific = c("3", "1"))
agreement(table, specific = c("3", "1", "2"))
agreement(table, specific = c(3))
agreement(table, specific = c("3", "1"), confint = TRUE)
agreement(df, specific = c("3", "1"), confint = TRUE)

iriseekhout/Agree documentation built on July 28, 2023, 11:24 p.m.