tests/testthat/helper-tiers.R

In tulpa: Templated Unified Library for Posterior Approximation in Bayesian Hierarchical Models

# Test tiers -- single source of truth for which cost level runs where.
#
#   Tier 1  structural   (no gate)           shape / class / dispatch / formula /
#                                             closed-form / finite-difference-gradient
#                                             checks; under 0.1 s; runs everywhere
#                                             including CRAN.
#   Tier 2  recovery      skip_on_cran()      one (or a few) fits to convergence,
#                                             parameter recovery vs a single simulated
#                                             truth, equivalence (dense == sparse,
#                                             serial == parallel, cpp == R). Seconds.
#                                             Runs by default locally + CI; skipped on
#                                             CRAN.
#   Tier 3  full          skip_if_not_slow()  MCMC / HMC sampler chains, multi-seed
#                                             coverage aggregates, multi-recovery loops.
#                                             Minutes. Opt-in; skipped by default
#                                             everywhere (including CRAN).
#
# Fast smoke mode: set TULPA_FAST=1 to collapse the suite to tier 1 only. Every
# tier-2 fit and tier-3 sampler then skips (reported as skips, never silently
# dropped), so a whole-suite plumbing / dispatch / closed-form run finishes in
# seconds. The default (TULPA_FAST unset) is unchanged, so the local dev loop,
# CI, and CRAN see exactly what they did before.
#
# Run profiles:
#   devtools::test()                                   # tiers 1 + 2  (default dev loop)
#   Sys.setenv(TULPA_FAST = "1"); test()               # tier 1 only  (fast smoke)
#   Sys.setenv(TULPA_SLOW_TESTS = "true"); test()      # tiers 1 + 2 + 3  (full validation)
#   R CMD check / CRAN                                 # tier 1 of the smoke files
#
# RELEASE GATE. Tier 3 holds the recovery / coverage / sampler-calibration
# evidence and is opt-in, so a routine test() or R CMD check runs plumbing only
# -- a green routine check says nothing about unbiasedness or interval coverage.
# Before any release or CRAN submission the full slow suite is a HARD gate, run
# green alongside the win-builder r-devel pass:
#
#   Sys.setenv(TULPA_SLOW_TESTS = "true", NOT_CRAN = "true"); devtools::test()
#
# One tier above that sits TULPA_FULL_RECOVERY=true: multi-hour full-strength
# sweeps (currently the joint SPDE NUTS 10-seed recovery in
# test-spde-nuts-joint.R) that additionally require TULPA_SLOW_TESTS. Not part
# of any routine profile; run before a release that touched the SPDE log-post /
# PC prior / NC transform.
#
# Put exactly one tier gate as the first line of any test_that() block whose cost
# is a model fit or an MCMC sample; leave structural blocks ungated so CRAN still
# exercises them. Fast mode rides the tier-2 / tier-3 gates, so a new fit placed
# behind skip_on_cran() / skip_if_not_slow() is collapsed by TULPA_FAST=1 with no
# extra wiring.
#
# The 0.1 s bound is what keeps the CRAN check inside its time budget: tier 1 is
# some 1700 blocks over 480 files, and the check farm runs it 1.3x slower than a
# desktop on top of a fixed per-file cost. A block that measures above it under
# the CRAN profile takes skip_on_cran() as its first line.
#
# CRAN runs a narrower slice of tier 1 than that bound admits. win-builder R-devel
# spends some 640 s on the whole check against CRAN's 600 s, and the tests are the
# share of it the package decides; so a tier-1 block that measures over 15 ms
# under the CRAN profile, on the lower of two runs, also skips on CRAN. It stays
# tier 1 everywhere else: its gate is skip_on_cran_budget(), which reads NOT_CRAN
# alone, so the dev loop and TULPA_FAST=1 both run it. Such blocks sit between
# the CRAN-run part of a file and its skip_on_cran() section, behind a top-level
# skip_on_cran_budget(), and each carries that gate as its first line as well.
#
# CRAN also runs only the smoke files, one per subsystem (CRAN_SMOKE_FILES
# below): tests/testthat.R hands cran_smoke_filter() to test_check(), so no other
# file is sourced there. Sourcing a file costs more than its tier-1 blocks do --
# the 245 files CRAN ran held 4 s of block time and took 43 s on win-builder --
# so the file count is what the budget reads. A smoke file is one whose tier-1
# part covers its subsystem's constructors and closed forms cheaply; the CI check
# matrix (TULPA_CRAN_BUDGET=off) and every other profile run all files.
#
# A skipped block is not free: test_that() sets up its test context (a temporary
# directory, the locale, the reproducible-output options) before the gate in the
# body runs, about 10 ms a block, and the suite gates over a thousand of them. A
# file none of whose blocks is tier 1 therefore also carries skip_on_cran() at
# top level, before its first expression, which skips the file in one step and
# reports it as one skip. The block gates stay, so moving a block to another
# file keeps it gated. A tier-1 block added to such a file goes in another file,
# or the file gate comes off.
#
# A file that holds tier-1 blocks as well puts them, and every top-level
# statement, first; a top-level skip_on_cran() follows, and the gated blocks
# come after it. CRAN then runs the tier-1 part in the order it is written and
# skips the rest in one step. A new gated block goes below that line, a new
# tier-1 block or fixture above it.
#
# A tier-1 block is under 0.1 s, and a structural identity swept over a large
# parameter grid is not. That is a fixture size rather than a tier: size the grid
# through cran_fixture() (below) instead of gating the block, so CRAN keeps
# reading the identity on a small grid and the dev loop keeps sweeping the whole
# one. Reach for a tier gate only when the block's cost is a fit or an
# equivalence, which are tier 2 whatever the grid.

# Fast smoke gate. Set TULPA_FAST=1 to keep only the ungated tier-1 structural
# tests; everything that fits a model or samples a chain skips.
skip_if_fast <- function() {
  if (identical(Sys.getenv("TULPA_FAST"), "1")) {
    testthat::skip("TULPA_FAST set: tier-1 structural smoke only")
  }
}

# Tier 3 full-validation gate: samplers, multi-seed coverage, multi-recovery
# loops. Fast mode takes precedence so TULPA_FAST=1 wins even if TULPA_SLOW_TESTS
# is also set.
skip_if_not_slow <- function() {
  skip_if_fast()
  if (!identical(tolower(Sys.getenv("TULPA_SLOW_TESTS")), "true")) {
    testthat::skip(
      "full-validation tier: set TULPA_SLOW_TESTS=true to run samplers / coverage loops"
    )
  }
}

# Tier 2 recovery gate: single-fit recovery / equivalence checks. Wraps the
# testthat builtin so all three tier gates have one definition here, and folds in
# fast smoke mode -- TULPA_FAST=1 skips tier 2 alongside CRAN. testthat::skip_on_cran()
# is the builtin (gated on the NOT_CRAN env var, which devtools sets locally and
# CRAN leaves unset).
skip_on_cran <- function() {
  skip_if_fast()
  testthat::skip_on_cran()
}

# CRAN time-budget gate: a tier-1 block over the CRAN budget (see the header).
# It skips on CRAN only, so TULPA_FAST=1 still runs the block, and
# TULPA_CRAN_BUDGET=off opens it under the CRAN profile too, which is how the CI
# check matrix runs the whole of tier 1 on every platform.
skip_on_cran_budget <- function() {
  if (!cran_budget_applies()) {
    return(invisible())
  }
  testthat::skip_on_cran()
}

# TRUE under the CRAN profile, unless TULPA_CRAN_BUDGET=off opens the budget.
cran_budget_applies <- function() {
  !identical(Sys.getenv("NOT_CRAN"), "true") &&
    !identical(tolower(Sys.getenv("TULPA_CRAN_BUDGET")), "off")
}

# The CRAN smoke set: one file per subsystem, named without "test-" and ".R".
CRAN_SMOKE_FILES <- c(
  "adjacency", "formula", "family-count-extensions", "prior-predict",
  "spatial-spde-api", "temporal", "nested-laplace-joint-multi", "beta-nuts",
  "methods-interop", "diagnostics", "posterior-sbc", "psis", "tgmrf",
  "control-check", "simulate", "autodiff-primitives", "re-cov-prior",
  "nested-posterior-draws", "criteria-tulpa-fit", "em-laplace",
  "pg-index-validation", "pg-nngp-conditional", "st-hsgp-prior", "mcar-prior",
  "ccd-modefind-budget", "auto-grid-provenance"
)

# test_check()'s file filter: the smoke set under the CRAN budget, else every file.
cran_smoke_filter <- function() {
  if (!cran_budget_applies()) {
    return(NULL)
  }
  paste0("^(", paste(CRAN_SMOKE_FILES, collapse = "|"), ")$")
}

# Tier-1 fixture size. A structural identity read over a grid of parameter
# combinations -- a closed form against quadrature, a score against finite
# differences, a normalizer summed over its support -- is tier 1 by nature, and
# what makes it cost seconds rather than milliseconds is the size of the grid,
# not the kind of work. Such a block reads the small grid where TRUE and the
# full one otherwise, so the identity is checked everywhere and the dev loop
# still sweeps every combination. Keyed on the same NOT_CRAN the tier-2 gate
# reads, so one environment decides both.
#
# This is the remedy for an oversized tier-1 block. A block whose cost is a FIT
# or an EQUIVALENCE takes a tier gate instead -- shrinking those would leave
# CRAN running a weaker version of a check that belongs to tier 2.
cran_fixture <- function() {
  !identical(Sys.getenv("NOT_CRAN"), "true")
}

# Publish the gates into the global environment. This is what makes the
# skip_on_cran() above take precedence over the identically named testthat
# builtin when a test file calls it bare: globalenv is searched ahead of the
# attached testthat package, whereas a helper-scoped binding is not. Mirrors the
# namespace-aliasing in helper-internal.R.
for (.nm in c("skip_if_fast", "skip_if_not_slow", "skip_on_cran", "skip_on_cran_budget",
              "cran_budget_applies", "cran_fixture")) {
  assign(.nm, get(.nm), envir = globalenv())
}
rm(.nm)