position_on_surface: Position an observed agreement coefficient on its...

In grassr: Context-Conditioned Reporting for Binary Rater Reliability

Position an observed agreement coefficient on its DGP-calibrated surface

Description

position_on_surface() is the Target-2 reporting primitive for the merged GRASS binary-rater-reliability paper. Given an observed coefficient value and the study design ⁠(pi_hat, k, N)⁠, it inverts the coefficient to an implied panel quality q_hat (rater operating quality on the Se = Sp diagonal under the clustered latent-class DGP) and evaluates the observed value against EVERY calibrated quality level at the matched design – a sweep – from which it derives three read-outs: the pooled percentile, the consistency band on quality, and the p(q) sweep profile.

Usage

position_on_surface(
  obs_value = NULL,
  metric,
  pi_hat = NULL,
  k = NULL,
  N = NULL,
  method = c("empirical", "delta"),
  surface_data = NULL,
  ratings = NULL,
  reference_type = c("fitted", "oracle"),
  ...
)

Arguments

obs_value	Numeric scalar. The observed agreement coefficient. Optional when ratings is supplied (auto-derived via compute_observed(metric, Y)).
metric	Character scalar. One of "pabak", "fleiss_kappa", "mean_ac1", "krippendorff_a", "icc".
pi_hat	Numeric scalar in ⁠(0, 1)⁠. The panel-identified marginal positive rate. Optional when ratings is supplied (auto-derived via mean(Y)); otherwise supply mean(Y) from the rating matrix.
k	Integer >= 2. Number of raters. Optional when ratings is supplied (auto-derived as ncol(Y)).
N	Integer >= 1. Number of subjects. Optional when ratings is supplied (auto-derived as nrow(Y)).
method	One of "empirical" (default; uses the bundled sim-derived empirical q_hat sampling distribution) or "delta" (closed-form normal approximation from delta-method SE).
surface_data	Optional. A list with one or more of the following components, used when method = "empirical": per_rep: a vector of per-rep metric values at the caller's own ⁠(q, pi_hat, k, N)⁠ cell – an empirical sampling distribution at that design. Honored for reproducibility audits; yields a plain cohort percentile with no sweep or consistency band. q_grid_per_rep, q_grid: legacy per-q-grid empirical inputs. Retained for backward compatibility but no longer consumed internally (the sweep convention consults the bundled per-cell quantile surface); supplying them draws a note. When surface_data is NULL and method = "empirical", the function uses the bundled empirical q_hat surface, falling back to the delta-method sweep when that surface is unavailable at the design.
ratings	Optional. From v0.2.0 this is the primary input for all metrics (not just ICC): supplying an ⁠N x k⁠ rating matrix auto-derives obs_value, pi_hat, k, and N. Accepts an ⁠N x k⁠ integer matrix of 0/1 values (rows = subjects, cols = raters), a data.frame with k rater columns, or a length-2 list of equal-length 0/1 vectors (k = 2). For metric = "icc", supplying ratings (also accepted as a long data.frame with columns subject and rating) additionally enables a glmer fit for ⁠(mu, tau2)⁠ that pins down the correct F_key for ICC inversion; without ratings, ICC falls back to a nearest-M1 F_key lookup with a prominent caveat note (tau2 is unidentified from pi_hat alone). The glmer path requires lme4 (Suggests).
reference_type	For metric = "icc" only. One of "fitted" (default; GLMM-gap-corrected reference matching what practitioners compute via glmer) or "oracle" (closed-form ⁠sigma^2_subject / (sigma^2_subject + pi^2/3)⁠ with ⁠sigma^2_subject⁠ known from F). Use "oracle" only if obs_value was computed via oracle variance decomposition (non-standard for applied work). For N beyond the fitted-reference sim range (currently N > 200), the function auto-falls-back to oracle with an explanatory note.
...	Reserved for future extension.

Details

The function implements the v0.7.1 sweep convention (ratified 2026-07-05): the practitioner cites the observed coefficient, its pooled percentile (position within the design's achievable range), and the consistency band on panel quality. q_hat is promoted to the card via the consistency band; it also carries the surface parameterization and delta-method SE. The stipulated four-band adjective (Poor/Moderate/Strong/Excellent) and the modal-band confidence qualifier (decisive/moderate/weak) are retired.

Value

A list of class grass_surface_position with fields:

• observed_value – echo of obs_value

• metric – echo of metric

• design – list(pi_hat, k, N)

• q_hat – implied panel quality (coefficient inverted on the reference curve); the point estimate the consistency band surrounds

• se_q_hat – delta-method SE of q_hat

• percentile – pooled percentile in ⁠[0, 1]⁠ (the print method and the card show it on the 0-100 scale): the observed coefficient's position within the design's full achievable range (trapezoid-weighted mixture over every calibrated quality level). Monotone in obs_value by construction.

• percentile_basis – provenance string for percentile

• band – 95% test-inversion consistency band on quality (printed as "consistency band"): list(lo, hi, level, open_low, open_high, note). The quality levels whose sampling distributions are consistent with the observed value at this design.

• sweep – data.frame(q, p): the full profile ⁠p(q) = P(coefficient <= obs_value | quality q, this design)⁠

• sampling_method – which method was used

• reference_used – which reference produced the curve

• notes – character vector of caveats (e.g. nearest-neighbor gaps)

The three read-outs of the sweep

At the matched design ⁠(F/pi_hat, k, N)⁠ the observed coefficient is evaluated against every calibrated quality level q with no cell selection and no q snapping. Write ⁠p(q) = P(coefficient <= obs_value | panel of quality q, this design)⁠.

• percentile – the pooled percentile: a trapezoid-weighted average of p(q) over the calibrated quality axis (trapezoid because the grid is non-uniform). It reads as the observed coefficient's position within the design's full achievable agreement range and is monotone in obs_value by construction. Returned in ⁠[0, 1]⁠; callers such as grass_report() print it on the ⁠[0, 100]⁠ scale. This replaces the retired nearest-q_hat-cell percentile, whose cohort selection by a statistic derived from the coefficient made it a non-monotone sawtooth (panel review 2026-07-05).

• band – the 95% test-inversion consistency band on quality: the quality levels q with ⁠0.025 <= p(q) <= 0.975⁠, endpoints interpolated where p(q) crosses 0.975 (lower) and 0.025 (upper). Open-ended at the grid boundary is reported with a boundary flag.

• sweep – the full data.frame(q, p) profile, the object the sweep-ridgeline graphic renders.

Sweep construction

Two methods are implemented.

Empirical method (method = "empirical", default): at the matched ⁠(F_key, k, N)⁠ cell of the bundled empirical_q_hat_surface, ranks q_hat within each calibrated quality cell's q_hat_rep distribution (monotone-equivalent to ranking obs_value within the cell's coefficient distribution). The full per-rep data is not bundled (~300 MB); the package ships a precomputed multi-point empirical-quantile summary per cell. The whole quality axis is consulted – there is deliberately no q selection. When the design ⁠(pi_hat, k, N)⁠ falls outside the simulated grid, nearest-neighbour clamping is applied and flagged in notes.

Delta method (method = "delta"): the summary-stats-only fallback (also used for ICC, whose reference curve carries the F-shape conditioning). At each swept q it approximates the sampling distribution as Normal(E[metric](q), sd_metric(q)) and evaluates ⁠p(q) = pnorm(obs_value; mean, sd)⁠ on the calibrated q axis. A caller-supplied surface_data$per_rep single-cohort vector is still honored for reproducibility audits, yielding a plain cohort percentile with no sweep or band.

Internal reference-surface arithmetic

Under the clustered latent-class DGP with symmetric raters (Se = Sp = q), the large-N closed forms for PABAK, Fleiss kappa, AC1, and Krippendorff's alpha depend on q and the marginal positive rate ⁠pi_+⁠ only. This function uses pi_hat as a plug-in for ⁠pi_+⁠ and inverts the observed value on a 501-point q-grid on ⁠[0.5, 1]⁠ (matching ⁠paper2/code/12_q_inversion.R⁠ resolution). ICC requires the full subject-prevalence distribution F; in the absence of surface_data containing an ICC lookup, ICC requests fall through to a warning-noted delta-method approximation using the caller-supplied q_hat_override / se_q_hat_override if present, or stop with a clear message.

Ratings-primary path

From v0.2.0, the preferred entry point is to hand the rating matrix directly: position_on_surface(ratings = Y, metric = "pabak"). When ratings is supplied, the function auto-derives obs_value (via compute_observed(metric, Y)), pi_hat (mean(Y)), k (ncol(Y)), and N (nrow(Y)); any of those four arguments still supplied by the caller wins. This collapses the audit-style scalar-input path used in v0.1.x to a single matrix argument while keeping the scalar path callable for reproducibility checks. ratings accepts an ⁠N x k⁠ integer matrix in ⁠{0, 1}⁠, a data.frame with k rater columns, or a length-2 list of equal-length 0/1 vectors (k = 2). Round-trip equality with the scalar path is a tested invariant.

See Also

check_asymmetry() for the companion Column A tier (rater asymmetry model-safety).

Examples

# Ratings-primary path: just hand it the matrix.
set.seed(1)
Y <- matrix(rbinom(1000, 1, 0.3), nrow = 200, ncol = 5)
position_on_surface(ratings = Y, metric = "pabak")

# Equivalent scalar-input path (audit):
position_on_surface(
  obs_value = 2 * mean(Y[, 1] == Y[, 2]) - 1,  # PABAK on first pair
  metric = "pabak", pi_hat = mean(Y), k = ncol(Y), N = nrow(Y)
)

# Scalar path -- the three read-outs of the sweep convention.
r <- position_on_surface(
  obs_value = 0.62,
  metric    = "pabak",
  pi_hat    = 0.42,
  k         = 5,
  N         = 50
)
r$percentile   # pooled percentile of the achievable range
r$band         # consistency band on panel quality
head(r$sweep)  # the full p(q) profile

# Fleiss kappa at imbalanced prevalence.
position_on_surface(
  obs_value = 0.18,
  metric    = "fleiss_kappa",
  pi_hat    = 0.08,
  k         = 3,
  N         = 200
)

grassr documentation built on Sept. 22, 2026, 5:08 p.m.