sfa_coverage: Audit the Content Validity of a Scale Against a Construct...

In semanticfa: Semantic Factor Analysis of Language Model Embeddings

Audit the Content Validity of a Scale Against a Construct Region

Description

Quantifies content validity as geometric overlap between a scale's items and a construct region built with sfa_build_region(). Two headline numbers decompose the audit into Messick's two threats, each named by what it scores: construct coverage (the fraction of construct texts with an item within the coverage radius; its complement is construct underrepresentation) and item relevance (the fraction of items passing a per-item test against the ideal-item null; its complement is construct-irrelevant variance).

Usage

sfa_coverage(
  items,
  region,
  definition = NULL,
  keep_frac = NULL,
  construct = NULL,
  factor = NULL,
  cross = FALSE,
  radius_q = 0.95,
  radius = NULL,
  alpha = 0.05,
  p_adjust = c("none", "BH"),
  n_draws = 20L,
  n_null = 200L,
  sense_gate = TRUE,
  min_silhouette = 0.65,
  trim = 0.25,
  screen_items = TRUE,
  overlap_threshold = 0.6,
  n_boot = 200L,
  max_gaps = 6L,
  gap_quotes = 3L,
  embed = "sbert",
  model = NULL,
  cache = TRUE,
  seed = 1L,
  delta_q = NULL,
  k_precision = NULL
)

Arguments

items	Character vector of item text, or a data frame with an item/text column (and optionally code), as in sfa(). When the data frame also has a factor column (subscale assignments), the audit runs per factor by default - one audit per (item set, construct claim) pair, which is the unit content validity is defined on - and returns an "sfa_coverage_battery". Use factor to restrict to a subset.
region	An "sfa_region" object from sfa_build_region(), the path to one saved with its file argument, or - for multi-factor audits - a list of regions named by factor (each factor is audited against its own construct region). A single region with a multi-factor scale audits every factor against that same region, with a message.
definition	Optional definition overriding the region's stored one - the construct-narrowing workflow. When supplied, the region is first restricted to the sentences the narrower definition explains best, then the usual filters apply with the new definition as seed. How the restriction is made depends on keep_frac (see below). Default NULL uses region$definition with no restriction.
keep_frac	How much of the region a supplied definition keeps. NULL (default) uses the comparative rule: keep the sentences more similar to the narrower definition than to the region's original one. That rule is strict - when the narrower definition is a close paraphrase of the original, sentences containing the construct term hug the original definition and the rule can keep almost nothing. A number in (0, 1] switches to rank narrowing: keep the keep_frac fraction of the region most similar to the narrower definition, which always yields a region of known size. Rank narrowing is the recommended workflow for testing a narrowed construct claim, usually together with a fixed radius from the un-narrowed audit (narrowing shrinks the region, and recalibrating on the shrunken region would change the yardstick along with the claim).
construct	Optional display label for the (possibly narrowed) construct. Default: the region's construct, or the first words of a supplied definition.
factor	Which factors to audit when the items carry factor assignments. Default NULL audits all of them. A character vector of factor names restricts the battery to that subset (a single name returns a plain "sfa_coverage" audit). With character-vector items lacking assignments, factor may instead be a vector of one assignment per item.
cross	Audit every factor against every region (requires factor assignments and a named region list)? Default FALSE. The result is an "sfa_coverage_cross" matrix of audits - the content analogue of a multitrait matrix: items should be relevant to their own construct's region (diagonal) and irrelevant to their siblings' (off-diagonal), which is discriminant content validity measured from item text alone. Off-diagonal relevance is floored by how separable the constructs are in language, not by zero - the printed output states this caveat. sfa_cross_matrix() extracts the numeric matrix.
radius_q	Null quantile defining the coverage radius. Default 0.95 (ideal scale covers ~95% of the region; each gap call is a test at alpha = 1 - radius_q).
radius	Optional fixed coverage radius, overriding the calibrated one - the anchoring workflow, the analogue of holding a test's cut score fixed when equating forms. Calibration ties the radius to the audited condition, so any comparison that changes the item set or the region (deleting items, auditing a short form, rank narrowing) would otherwise move the yardstick along with the thing being measured: deleting items grows the calibrated radius and can raise coverage. For such comparisons, take ⁠$radius⁠ from the reference audit and pass it here so every condition is scored against the same yardstick. The item-relevance null is drawn at the fixed radius, and bootstrap resamples hold it fixed rather than recalibrating. The calibrated radius is still computed and returned as ⁠$radius_calibrated⁠ for comparison. Default NULL calibrates as usual.
alpha	Per-item test level against the ideal-item null. Default 0.05. An item is flagged when its corroboration count's empirical p-value is at most alpha.
p_adjust	Multiplicity handling for the item flags: "none" (per-item tests, default) or "BH" (Benjamini-Hochberg false discovery rate across the scale's items: of the flagged items, at most alpha are expected to be false alarms).
n_draws	Draws for the matched-size null behind the radius. Default 20.
n_null	Draws for the ideal-item count null behind the p-values (more draws give finer p resolution). Default 200.
sense_gate	Apply the sense gate (2-means on seed similarity; drops the wrong-sense/incidental-mention mode only when the split is real)? Default TRUE.
min_silhouette	Minimum 1-d silhouette for the sense gate to fire. Default 0.65 (a unimodal 1-d Gaussian scores ~0.55 under a forced 2-means split; genuine sense mixtures score higher).
trim	Incidental-mention trim: fraction of the region with the lowest similarity to the definition to drop before auditing (web corpora carry a tail of sentences that merely mention the term). Applied after the sense gate and reported in the print output. Default 0.25; set 0 to disable.
screen_items	Drop region sentences that near-duplicate the audited items (circularity rule)? Default TRUE.
overlap_threshold	Item-screen lexical threshold: a region sentence is dropped when at least this fraction of its content words appears in a single item (default 0.6), or - the self-calibrating geometric criterion, no constant - when it is more similar to an item than to any other region sentence (a leaked item copy always is; an on-topic corpus sentence never is; fixed cosine thresholds do not transfer across encoders' similarity scales).
n_boot	Bootstrap resamples of the region for percentile confidence intervals (0 to skip). Default 200.
max_gaps	Maximum gap clusters to report. Default 6.
gap_quotes	Example sentences quoted per gap. Default 3.
embed, model, cache	Passed to sfa_embed() for the items and seed. model defaults to the region's encoder; overriding it is almost always a mistake (the audit compares points in one space) and produces a warning.
seed	Random seed for the null draws, gap clustering, and bootstrap. Default 1.
delta_q, k_precision	Deprecated (pre-0.3.0 names). delta_q is mapped to radius_q with a warning. k_precision is ignored with a warning: the fixed-count rule it set was region-size dependent (sampling more construct text inflated relevance) and is replaced by the calibrated per-item test.

Details

One 95% convention calibrates both numbers against an ideal same-length scale - items drawn from the construct region itself. The coverage radius is the radius_q (default 95%) quantile of the matched-size null's nearest-neighbor distances, so an ideal scale's construct coverage is about 0.95 at any scale length, region size, or embedding dimension. Each item's corroboration count (construct texts within its radius) gets an empirical p-value against the ideal-item null; items are flagged at alpha (default .05), so an ideal scale's item relevance is also about 0.95. The identity behind the convention: 1 - radius_q and alpha are per-decision Type I error rates of Monte Carlo tests. Because the null's counts grow with region size, the critical count rescales automatically - sampling more construct text cannot inflate relevance.

The printed report frames the two remedies for low coverage explicitly: add items aimed at the named gaps (each gap is labeled with distinctive terms and quoted example sentences from the region - real corpus content the scale does not reach), or narrow the construct claim (the covered subregion's distinctive terms describe what the items actually measure). Narrowing is cheap to test: re-run sfa_coverage() on the same region with a narrower definition (for example "academic procrastination") and a keep_frac, which restricts the region at audit time - no new extraction. Hold radius at the un-narrowed audit's value so the narrowed claim is scored against the same yardstick (see the two parameters below).

Value

An object of class "sfa_coverage" with the audit numbers, per-item corroboration counts and p-values, filter accounting, gap report, and provenance - or, when the items carry factor assignments and more than one factor is audited, an "sfa_coverage_battery": a named list of "sfa_coverage" audits, one per factor (x$Depression is that factor's full audit). print() gives the report (a compact per-factor table for batteries); plot() draws the proportional-overlap diagram (type = "coverage"), the per-item relevance chart (type = "relevance"), or the coverage curve against the matched-size null (type = "curve"); sfa_gaps() returns the gap table.

Examples

## Not run: 
region <- sfa_load_region("procrastination_region.rds")
audit  <- sfa_coverage(my_items, region)
audit                      # coverage, relevance, gaps, the two remedies
plot(audit)                # proportional-overlap (Euler) diagram
plot(audit, type = "relevance")   # per-item counts, p-values, flags

# a multidimensional battery: items with a 'factor' column audit
# per subscale by default, each against its own construct region
battery <- sfa_coverage(dass_items,
                        region = list(Depression = reg_dep,
                                      Anxiety    = reg_anx,
                                      Stress     = reg_str))
battery                    # one row per factor
battery$Depression         # full report for one subscale
sfa_coverage(dass_items, reg_dep, factor = "Depression")  # just one

# test a narrower claim against the same region - no new extraction.
# Rank narrowing keeps the half of the region the narrower definition
# explains best; the fixed radius anchors the comparison to the full
# audit's calibration.
sfa_coverage(my_items, region, keep_frac = 0.5, radius = audit$radius,
             definition = "Academic procrastination is the delay of
                           study-related tasks despite expecting costs.")

# the same anchoring applies to any item-set comparison, e.g. a
# deletion study: score the reduced scale at the full scale's radius
sfa_coverage(my_items[-drop_idx, ], region, radius = audit$radius)

## End(Not run)

semanticfa documentation built on Sept. 2, 2026, 1:07 a.m.