check_asymmetry: Cross-coefficient panel asymmetry diagnostic

In grassr: Context-Conditioned Reporting for Binary Rater Reliability

Cross-coefficient panel asymmetry diagnostic

Description

check_asymmetry() takes an N x k binary ratings matrix and returns a scalar delta_hat (in quality percentage points, "pp"): the max-min spread of the implied panel qualities across the three agreement coefficients (PABAK, mean AC1, Fleiss kappa). Each coefficient inverts to its own q_hat on the shared (q, pi_+) reference; if the calibration DGP held exactly, all three would imply the same quality, so the spread measures cross-coefficient model discordance in interpretable units of quality. ICC never enters delta_hat, since its reference is distribution-sensitive in ways the agreement family is not.

Usage

check_asymmetry(ratings, axis = c("inter", "intra"), fit_icc = TRUE, ...)

Arguments

ratings	User input: an ⁠N x k⁠ binary matrix, an N x k data.frame whose columns are 0/1 / logical / 2-level factor, or a list of two equal-length 0/1 vectors (k = 2 paired form); the Quick start section of vignette("grassr") lists the accepted shapes.
axis	"inter" (default) or "intra". Selects the surface family.
fit_icc	If FALSE, skip the lme4::glmer fit behind icc and drop ICC from the panel. icc never enters delta_hat, and the fit draws no random numbers, so a caller that reports only delta_hat and the implied qualities gets identical results at a fraction of the cost. The Monte Carlo null loop sets this; interactive users should not.
...	Forwarded to position_on_surface() (e.g. reference_type).

Details

delta_hat is a split-bias detector. It fires when raters tilt in different directions across (Se, Sp) – e.g., one rater high-Se / low-Sp, another high-Sp / low-Se – because the three coefficients respond to heterogeneous per-rater behavior differently and end up implying different panel qualities. The framework's other failure mode, shared/uniform bias (every rater tilts the same direction, e.g., a panel trained on one protocol all favoring specificity over sensitivity), produces uniform degradation across coefficients: small delta_hat, low implied quality. Shared bias is detected by the panel's implied quality (and its consistency band), not by delta_hat. A divergent flag therefore identifies a specific kind of disagreement – cross-coefficient inversion from heterogeneous rater behavior – and routes the user to the per-rater pairwise PABAK matrix and pooled-reference (Se_tilde, Sp_tilde) diagnostic.

Each coefficient is positioned on its reference surface via position_on_surface(), which reports its implied q_hat. The panel diagnostic is delta_hat = max(q_hat) - min(q_hat) (in pp of quality), computed over agreement-family coefficients whose observed value falls within the achievable range of their reference surface (see Surface-envelope clamp below).

Value

An S3 object of class grass_asymmetry_panel with fields:

• delta_hat: scalar implied-quality spread, in pp of quality

• delta_percentile: delta_hat's percentile on the matched (k, N, q_hat) null ECDF (NA if the null is uncalibrated at the design)

• flag: one of "aligned", "caution", "divergent"

• matched_null: list describing the matched null cell (k, N, q, prev — the bridged true-prevalence estimate the lookup conditioned on, prev_apparent — the panel's raw positive rate, prev_bridged, q_hat_panel, n_draws, snapped, interpolated, unstable_tail), or NULL if uncalibrated

• thresholds: named numeric vector of the implied (caution, divergent) pp cuts (95th/99th of the matched null)

• thresholds_source: one of "matched_null_ecdf", "not_applicable_k2", "not_calibrated"

• panel: data.frame with coefficient, observed, implied_q, percentile_pp (pooled percentile), clamped, in_delta_hat

• notes: character vector of unique caveats from the underlying surface positioning calls (e.g. nearest-neighbor gaps, ICC unavailability, matched-null provenance)

Surface-envelope clamp (v0.2.1+)

If an observed coefficient value falls outside the achievable range of its reference surface at the study's design ⁠(pi_hat, k, N)⁠, the inversion to q_hat clamps to the boundary. Including such clamped implied qualities in the max-min delta_hat would inflate the panel spread purely because of the clamp, not because the panel disagrees on quality. Since v0.2.1 the function therefore excludes clamped coefficients from delta_hat whenever at least two unclamped agreement-family coefficients remain. The affected coefficients are still shown in the returned panel data.frame with clamped = TRUE, and a note in ⁠$notes⁠ names which coefficients were excluded. This matters most often for ICC (which never enters delta_hat anyway) and at designs beyond the bundled reference range, where a coefficient clamps to the achievable boundary. If fewer than two unclamped agreement-family coefficients remain, delta_hat falls back to the raw spread including clamped values and the note records the fallback.

Flag from the matched null (v0.7.0/0.7.1)

The per-(k, N) size-alpha threshold table is retired. The flag is delta_hat's percentile on the null distribution of delta_hat at the matched (k, N, q_hat) cell of the bundled delta_null_ecdf, with the cut convention >= 95th caution, >= 99th divergent. The panel's q_hat is resolved first (median of the agreement-family implied qualities), then the matched null cell is looked up; the reported thresholds carry the implied pp cuts (95th/99th of that null) as context, and thresholds_source records how the flag was resolved. The three flags are:

• aligned (below the 95th percentile of the matched null): the panel agrees on the implied quality. Any single coefficient is a stable summary; the primary coefficient (Table 2) carries the headline.

• caution (>= 95th, < 99th): the panel is mildly inconsistent. Report the primary coefficient with a caution flag and the delta_hat value.

• divergent (>= 99th): no single coefficient is a stable summary. Use latent_class_fit() to recover per-rater ⁠(Se, Sp)⁠ and report those instead.

The new check_asymmetry(ratings, ...) signature replaces the v0.1.x check_asymmetry(se, sp, ...) per-rater signature. For Calling check_asymmetry() with the pre-0.2.0 se = ... and sp = ... arguments is an error; per-rater sensitivity and specificity come from latent_class_fit().

See Also

latent_class_fit() for the divergent-branch recovery of per-rater ⁠(Se, Sp)⁠; position_on_surface() for the underlying surface positioning.

Examples

set.seed(1)
# Build a 5x200 symmetric panel -- should print as 'aligned'.
Y <- matrix(rbinom(5 * 200, 1, 0.30), nrow = 200, ncol = 5)
check_asymmetry(Y)

grassr documentation built on Sept. 22, 2026, 5:08 p.m.