grass_power: Plan a rating study: what a design can show

In grassr: Context-Conditioned Reporting for Binary Rater Reliability

Plan a rating study: what a design can show

Description

grass_power() plans a study's size. Assume a prevalence and a rough rater quality, and it says what a design of a given size can show about the panel, and how many more subjects or raters it would take to show more. The result describes the design, how much a study of that size can learn about its raters.

Usage

grass_power(
  metric,
  q = NULL,
  q0 = NULL,
  target = NULL,
  pi_hat = NULL,
  prevalence = NULL,
  k = NULL,
  N = NULL,
  power = NULL
)

Arguments

metric	One of "pabak", "fleiss_kappa", "mean_ac1", "icc".
q	Panel quality, the probability of a correct call on the Se = Sp diagonal, in ⁠[0.55, 0.99]⁠.
q0	The lower edge of the quality resolution the study is planned for, in ⁠[0.55, 0.99]⁠: the study is sized to show that a panel of quality q is above q0. Give q0 or target, not both.
target	A fixed coefficient value to reach. Give q0 or target, not both.
pi_hat	Observed positive rate, the share of all ⁠N x k⁠ ratings that are positive, in ⁠[0.05, 0.95]⁠. This is what the card measures and what the surfaces are indexed by. Give pi_hat or prevalence, not both.
prevalence	True positive rate of the finding, in ⁠[0.01, 0.99]⁠. A planner usually has this rather than pi_hat. Under the symmetric reference model a panel of quality q turns it into pi_hat = prevalence * q + (1 - prevalence) * (1 - q), and the function does that conversion, at each candidate q when q is the one being solved. The result carries both rates. A pi_hat that no panel of quality q can produce at any prevalence is refused.
k	Number of raters. Snaps to the nearest calibrated rater count (2, 3, 5, 8, 15, 25), as the surfaces do everywhere.
N	Number of subjects in ⁠[15, 1000]⁠.
power	Probability, in ⁠(0, 1)⁠.

Details

A study sized to show that a panel of quality q is above q0 returns a 95% consistency band on quality whose lower edge lies above q0. q0 sets the resolution of the plan. A panel weaker than assumed returns a band that is lower and about as wide, so the study reports the quality it finds at the precision it was planned for. The convention follows stats::power.t.test(). Fix four of q, the positive rate (prevalence or pi_hat), k, N, power, leave one NULL, and the function solves for it.

Whether more subjects or more raters raises power depends on prevalence. At balanced prevalence a few more raters do the work of subjects. At a rare or very common finding they do not, because a small sample holds only a handful of the minority class, and the answer is more subjects.

The function also accepts target, a fixed coefficient value, for a threshold imposed from outside (a journal's or regulator's band). A fixed coefficient value means something different at each design. When the value a panel of quality q produces at the design is below target, power falls with N. Give q0 or target, not both.

Value

An object of class grass_power: the design quantities with the solved one filled in, solved naming it, mode ("quality" or "value"), feasible, reason (when not feasible), expected (the median coefficient a panel of quality q produces at the design, in "value" mode), curve (power across the solved variable's range, the data plot() draws), and notes from the surface lookup.

How it is computed

Every quantity is a direct read of the quality sweep that position_on_surface() returns, p(q) = P(coefficient <= c | q, design). Nothing is simulated at call time. For q0, by test inversion the lower end of the band is above q0 exactly when the observed coefficient exceeds the 97.5th percentile of the q0 distribution, c0, so power = 1 - p_q(c0). For target, power = 1 - p_q(target).

Solving for N or k

The smallest value on the calibrated surface at which power is reached. Below about 25 subjects the curve can step or dip between adjacent sample sizes, because the coefficient takes few distinct values there. When none reaches it the result carries NA, feasible = FALSE, and the reason, including the best power any design on the surface reaches.

Solving for the positive rate

Leave both pi_hat and prevalence NULL. The result is the range of observed positive rates over which power is reached, solution holds its two endpoints, and prevalence holds the same range converted at q.

Examples

# Raters assumed near quality 0.90, prevalence 10%, three raters: how
# many subjects to show the panel is above quality 0.80, 80% power?
pw <- grass_power("fleiss_kappa", q = 0.90, q0 = 0.80, prevalence = 0.10,
                  k = 3, power = 0.80)
pw
if (requireNamespace("ggplot2", quietly = TRUE)) plot(pw)
# Leave `prevalence` out to get the range of positive rates over which
# a fixed design keeps that power; give `target =` instead of `q0` to
# size against a fixed coefficient value. Both are shown in the vignette.

grassr documentation built on Sept. 22, 2026, 5:08 p.m.