calculate_agreement: Calculate inter-rater agreement statistics beween team...
In nthun/metamanager: Manage Meta-analysis Workflow in R

Description Usage Arguments Value See Also Examples

Calculate agreement summaries and Krippendorff's alpha for all reviewer pairs. Krippendorff's alpha can be calculated for nominal, ordinal, interval, and ratio scale variables.

1	calculate_agreement(df, decision_col = "decision", kr_method = "nominal")

`df`	a data frame that contains the screened articles, and having a name<chr> column
`decision_col`	variable name <chr> that defines the decision column (0/1)
`kr_method`	The scale of the decision variable. Can be nominal (default), ordinal, interval, or ratio

A data frame containing agreement summaries, and the Krippendorff's alpha statistic for all pairs of screeners. invalid_decision<int> is the total number of decisions that are not 1 or 0.

kripp.alpha

library(dplyr)
team_df <- tibble(name = c("Tom", "Jerry", "Rose"),
                  screening_effort = c(.5, .4, .1))

set.seed(1)
df <-
    workaholism_pubmed %>%
    make_id(identifier = c("doi", "pmid")) %>%
    assign_articles(team_df = team_df,
                    reviewers = 2,
                    effort_col = "screening_effort") %>%
    # Generate fake screening data
    mutate(decision = sample(c(0:1, NA_integer_),
                             prob = c(.6,.3, .1),
                             size = nrow(.),
                             replace = TRUE))

calculate_agreement(df, decision_col = "decision")