ICCrestricted: Range-restricted reliability with intra-class correlation

In patriciamar/ShinyItemAnalysis: Test and Item Analysis via Shiny

Range-restricted reliability with intra-class correlation

Description

Function estimating reliability with intra-class correlation for the complete or for the range-restricted sample.

Usage

ICCrestricted(
  Data,
  case,
  var,
  rank = NULL,
  dir = "top",
  sel = 1,
  nsim = 100,
  ci = 0.95,
  seed = NULL
)

Arguments

Data	matrix or data.frame which includes variables describing ID of ratees (specified in case), ratings (specified in var), and (optionally) rank of ratees (specified in rank).
case	character: name of the variable in Data with ID of the ratee (subject or object being evaluated, such as a respondent, proposal, patient, applicant etc.)
var	character: name of the variable in Data with the ratings/scores.
rank	numeric: vector of ranks of ratees. If not provided, rank of ratee is calculated based on average rating based on var variable.
dir	character: direction of range-restriction, available options are "top" (default) or "bottom". Can be an unambiguous abbreviation (i.e., "t" or "b").
sel	numeric: selected number (given > 1) or percentage (given <= 1) of ratees. Default value is 1 (complete dataset).
nsim	numeric: number of simulations for bootstrap confidence interval. Default value is 100.
ci	numeric: confidence interval. Default value is 0.95.
seed	seed for simulations. Default value is NULL, random seed. See lme4::bootMer() for more detail.

Value

A data.frame with the following columns:

n_sel	number of ratees selected/subsetted.
prop_sel	proportion of ratees selected.
dir	direction of range-restriction. NA if range is effectively not restricted (100% used).
VarID	variance due to ratee, "true variance", between-group variance.
VarResid	residual variance.
VarTotal	total variance.
ICC1	single-rater inter-rater reliability.
ICC1_LCI	lower bound of the confidence interval for ICC1.
ICC1_UCI	upper bound of the confidence interval for ICC1.
ICC3	multiple-rater inter-rater reliability.
ICC3_LCI	lower bound of the confidence interval for ICC3.
ICC3_UCI	upper bound of the confidence interval for ICC3.

Author(s)

Patricia Martinkova
Institute of Computer Science of the Czech Academy of Sciences
[e-mail]

Jan Netik
Institute of Computer Science of the Czech Academy of Sciences
[e-mail]

References

Erosheva, E., Martinkova, P., & Lee, C. (2021a). When zero may not be zero: A cautionary note on the use of inter-rater reliability in evaluating grant peer review. Journal of the Royal Statistical Society - Series A. Accepted.

Erosheva, E., Martinkova, P., & Lee, C. (2021b). Supplementary material for When zero may not be zero: A cautionary note on the use of inter-rater reliability in evaluating grant peer review.

Examples

# ICC for the whole sample
ICCrestricted(Data = AIBS, case = "ID", var = "Score", rank = "ScoreRankAdj")

# ICC for the range-restricted sample considering 80% of top ratees
ICCrestricted(
  Data = AIBS, case = "ID", var = "Score", rank = "ScoreRankAdj",
  sel = 0.8
)

patriciamar/ShinyItemAnalysis documentation built on May 18, 2023, 4:34 p.m.