sfa_redundancy: Detect Redundant (Near-Duplicate) Items

In semanticfa: Semantic Factor Analysis of Language Model Embeddings

Detect Redundant (Near-Duplicate) Items

Description

Finds pairs of items that are so semantically similar they are effectively duplicates — they add length without adding information. This is distinct from sfa_simplify, which removes weak items (far from their construct); redundancy targets near-twin items (very close to each other).

Usage

sfa_redundancy(x, threshold = NULL, method = c("wto", "cosine"))

Arguments

x	An "sfa" object (uses its similarity matrix) or a symmetric numeric item-by-item similarity matrix.
threshold	Redundancy cutoff. Item pairs with overlap at or above this value are flagged. Defaults to 0.25 for "wto" (the Unique Variable Analysis cut-off) and 0.80 for "cosine".
method	Overlap measure: "wto" (default) Unique Variable Analysis (Christensen et al. 2023): absolute weighted topological overlap on an EBICglasso network, the paper's estimator. Requires the EGAnet package. Because an embedding similarity matrix has no response sample, the network is estimated with a nominal sample size large enough to keep the EBIC model selection in its stable regime (EBIC over-shrinks to an empty graph when the sample size equals the item count). Estimating a sparse network first is what gives wTO its discriminating power; computing it on the dense matrix compresses every pair into a narrow band. "cosine" Direct pairwise similarity. Dependency-free and well spread for dense embedding matrices.

Value

An object of class "sfa_redundancy": a list with the flagged pairs (data frame: item_i, item_j, overlap), redundant clusters (connected groups of mutually redundant items), and suggest_remove (all-but-one item per cluster — keep one representative). Unique Variable Analysis is a detection method: Christensen et al. (2023) leave the handling of flagged redundancies to the researcher, so the keep-the-most-central-item suggestion (highest mean absolute similarity) is this package's convenience rule, not part of UVA.

References

Christensen, A. P., Garrido, L. E., & Golino, H. (2023). Unique Variable Analysis: A network psychometrics method to detect local dependence. Multivariate Behavioral Research, 58(6), 1165–1182. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1080/00273171.2023.2194606")}

See Also

sfa_simplify

Examples

data(big5)
fit <- sfa(
  data.frame(code = big5$codes, item = big5$items,
             factor = big5$factors, scoring = big5$scoring),
  embeddings = big5$embeddings, scoring = big5$scoring, nfactors = 5)

# flag near-duplicate item pairs
sfa_redundancy(fit, threshold = 0.8, method = "cosine")

semanticfa documentation built on Sept. 2, 2026, 1:07 a.m.