bias_variance: Estimate Bias and Variance

In rtemis: Schema-Driven Machine Learning and Visualization

Estimate Bias and Variance

Description

Decompose a learner's error into the part that comes from being systematically wrong and the part that comes from being unstable, by refitting it over many resamples of the training set and watching what its predictions do at a fixed test set.

Usage

bias_variance(
  x,
  hyperparameters,
  dat_test = NULL,
  true_values = NULL,
  resampler_config = NULL,
  test_p = 0.3,
  execution_config = setup_FutureExecution(),
  verbosity = 1L
)

Arguments

x	tabular data: Full dataset, outcome in the last column. Resamples are drawn from it after the test set is held out.
hyperparameters	Hyperparameters object: Must be fixed, not a search space – this measures one model, not a tuning run.
dat_test	Optional tabular data: Test set. NULL holds one out of x.
true_values	Optional Numeric vector: The true function at the test cases, in their order. Regression: f(x). Classification: the probability of the positive class. NULL measures bias against the observed outcome, which includes irreducible noise.
resampler_config	Optional ResamplerConfig object: How training sets are drawn. NULL uses 100 stratified bootstraps.
test_p	Numeric (0, 1): Fraction of x held out as the test set. Ignored when dat_test is given.
execution_config	ExecutionConfig object: Supplies the seed and the workers the resamples are fitted over.
verbosity	Integer: If > 0, print messages.

Details

For squared-error loss, the expected error at a point decomposes as

E[(y - \hat{f}(x))^2] = \sigma^2 + (E[\hat{f}(x)] - f(x))^2 + Var[\hat{f}(x)]

– irreducible noise, squared bias, and variance. This function estimates the second and third per test case, by fitting the model on n_resamples resamples of the training set and reading the spread and the average of the predictions each makes at the same test cases.

The test set is held out once and every resample is drawn from what is left, which is what makes the predictions comparable across resamples: a test set that moved would confound the model's variability with the sample's.

Value

BiasVariance object.

Whether the bias is really the bias

Squared bias is measured against true_values when you have them, and against the observed outcome when you do not. The difference is not cosmetic:

With true_values

bias_squared is the bias, exactly. Only simulated data has this, which is what makes a simulation study the place to ask how a learner divides its error.

Without

The observed outcome carries the irreducible noise, so what is reported is bias^2 + \sigma^2 and no amount of resampling separates them. It stays comparable between learners on the same data – they share the same \sigma^2 – so a ranking is still meaningful even though the level is inflated.

What the numbers are, exactly

variance is the sample variance across resamples (dividing by n_resamples - 1), which is unbiased for the variance of the fitted function. bias_squared is the plug-in estimate, and is inflated by roughly variance / n_resamples because the mean prediction is itself estimated – so prefer many resamples over few, and treat a bias_squared of the same order as variance / n_resamples as indistinguishable from zero.

Classification

The decomposition above is for squared error, and 0-1 loss does not decompose additively into bias and variance. For a binary outcome this function therefore decomposes the squared error of the predicted probability of the positive class, which does. true_values is then the true probability, not the class. Multiclass is not supported.

Author(s)

EDG

References

Geman S, Bienenstock E, Doursat R (1992). Neural Networks and the Bias/Variance Dilemma. Neural Computation, 4(1), 1-58. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1162/neco.1992.4.1.1")}

Examples

set.seed(2026)
n <- 200L
features <- data.frame(a = rnorm(n), b = rnorm(n))
dat <- data.frame(features, y = 2 * features[["a"]] + rnorm(n))
bv <- bias_variance(
  dat,
  hyperparameters = setup_CART(),
  resampler_config = setup_StratBoot(n_resamples = 10L),
  verbosity = 0L
)
bv

rtemis documentation built on Oct. 20, 2026, 1:06 a.m.