run_backtest_leaderboard: Multi-model rolling-origin backtest leaderboard

In forecastsuite: Multi-Model Time Series Forecasting Toolkit and Local App

Multi-model rolling-origin backtest leaderboard

Description

Runs build_cv_folds() once and refits every model in model_keys across every fold, ranking them by mean test MASE – a real multi-model extension of single-model cross-validation. A fold a model fails on is dropped for that model rather than failing the whole row; a model with zero successful folds (including an invalid model_keys entry) gets an NA row instead of being dropped, so it's still visible in the result as "failed."

Usage

run_backtest_leaderboard(
  model_keys,
  df,
  date_agg,
  horizon_periods,
  k_requested,
  build_args = function(model_key, train_df) {
     list(train_df = train_df, date_agg =
    date_agg)
 },
  window = c("expanding", "rolling"),
  train_window = NULL
)

Arguments

model_keys	Character vector of registry keys (see list_models()) to include in the leaderboard.
df	A data frame with ds/y columns – the series to backtest.
date_agg	Aggregation frequency (e.g. "day", "month"), passed through to each model's fit().
horizon_periods	Integer, each fold's test window length (see build_cv_folds()).
k_requested	Integer, the number of folds requested (may return fewer; see build_cv_folds()).
build_args	A function ⁠function(model_key, train_df)⁠ returning the full argument list for that model's fit() call (train_df/ date_agg included). The default passes train_df/date_agg through with no model-specific tuning; callers wanting each model's currently-configured parameters (e.g. the bundled app, which passes its own build_fit_args()) can supply their own.
window	"expanding" (the default) or "rolling", passed through to build_cv_folds() – see there for what each means.
train_window	Integer or NULL, passed through to build_cv_folds() when window = "rolling".

Value

A tibble with one row per model in model_keys, columns model (label), key, n_folds, mean_mase, sd_mase, mean_smape, mean_rmse, ranked ascending by mean_mase (NA last).

Examples

df <- data.frame(ds = as.Date("2024-01-01") + 0:59,
                  y = 100 + seq_len(60) * 0.2 + stats::rnorm(60, sd = 2))

run_backtest_leaderboard(c("arima", "ets"), df, date_agg = "day",
                          horizon_periods = 5, k_requested = 2)

forecastsuite documentation built on Oct. 18, 2026, 5:08 p.m.