simulateIRR: Calculates intraclass correlations (ICC) for simulated...

In jbryer/IRRsim: Simulate Inter-Rater Reliability Statistics

Calculates intraclass correlations (ICC) for simulated samples of raters and evaluations.

Description

This function wraps the [IRRsim::simulateRatingMatrix()] function to estimate inter-rater reliability statistics across many simulated rating matrices. It returns a 'list' with all the runs but can be converted to a data frame using the 'as.data.frame()' function. By default this function will run the simulations in parallel using the 'parallel' package using one less the number of available cores. Set 'parallel = FALSE' to run the simulations on one thread.

Usage

simulateIRR(
  nRaters = c(2),
  nRatersPerEvent = nRaters,
  nLevels = 4,
  nEvents = 100,
  nSamples = 100,
  agreements = seq(0.1, 0.9, by = 0.1),
  response.probs = rep(1/nLevels, nLevels),
  showShinyProgress = FALSE,
  showTextProgress = !showShinyProgress,
  numCores = (parallel::detectCores() - 1),
  parallel = (numCores > 1),
  ...
)

Arguments

nRaters	the number of available raters
nRatersPerEvent	the number of ratings for each per scoring event.
nLevels	the number of possible outcomes there are for each rating.
nEvents	the number of rating events within each matrix.
nSamples	the number of sample matrices to estimate at each agreement level.
agreements	vector of percent agreements to simulate.
response.probs	probability weights for the distribution of scores. See [IRRsim::simulateRatingMatrix()] for more information.
showShinyProgress	show progress bar as simulations are generated.
showTextProgress	show progress bar as simulations are generated.
numCores	number of cores to use if the simulation is run in parallel.
parallel	whether to simulated the data using multiple cores.
...	currently not used.

Details

For reproducibility using the [base::set.seed()] function be sure to set 'parallel = FALSE'.

Value

a list of length nSamples * length(nRaters) * length(agreements). Each element of the list represents one simulation with the following values:

k

the number of raters used in the simulation.

simAgreement

the calculated percent agreement from the sample.

agreement

the specified percent agreement used for drawing the random sample.

skewness

skewness of all responses.

kurtosis

Kurtosis for all responses.

MaxResponseDiff

the difference between the most and least freqeuent responses.

ICC1

ICC1 as described in Shrout and Fleiss (1979)

ICC2

ICC2 as described in Shrout and Fleiss (1979)

ICC3

ICC3 as described in Shrout and Fleiss (1979)

ICC1k

ICC1k as described in Shrout and Fleiss (1979)

ICC2k

ICC2k as described in Shrout and Fleiss (1979)

ICC3k

ICC3k as described in Shrout and Fleiss (1979)

Fleiss_Kappa

Fleiss' Kappa for m raters as described in Fleiss (1971).

Cohen_Kappa

Cohen's Kappa as calculated in psych::cohen.kappa. Note that this calculated for all datasets even though it is only appropriate for two raters.

data

The simulated matrix

See Also

as.data.frame.IRRsim

Examples

icctest <- simulateIRR(nLevels = 3,
                       nRaters = 2,
                       nSamples = 10,
                       parallel = FALSE,
                       showTextProgress = FALSE)
summary(icctest)

jbryer/IRRsim documentation built on April 23, 2023, 1:58 a.m.