run_comprehensive_erc_analysis: Run Comprehensive Forecast Evaluation Analysis

In RCtest: Reality Check and Predictive Ability Tests for Forecast Evaluation

Run Comprehensive Forecast Evaluation Analysis

Description

Runs a full suite of reality check and density forecast evaluation tests on one or more datasets. Tests included: White's Reality Check (WRC), Superior Predictive Ability (SPA), Conditional Predictive Ability (CPA), ZP Quantile Loss test, Kullback-Leibler Information Criterion (KLIC) test, Kupiec Unconditional Coverage (UC) test, CRPS-based CDF comparison, and per-model Diebold-Mariano statistics across four error metrics (MSE, MAE, MASE).

Technical Abbreviations:

• WRC: White's Reality Check (White, 2000). Tests whether any competing forecast has lower expected loss than the benchmark; controls family-wise error rate.

• SPA: Superior Predictive Ability test (Hansen, 2005). A studentized extension of WRC with improved power that corrects for irrelevant forecasts.

• CPA: Conditional Predictive Ability test (Giacomini & White, 2006). Tests whether loss differentials are predictable by a conditioning variable.

• ZP: Quantile Loss test (Corradi & Swanson, 2006). Evaluates whether any competing forecast better calibrates the probability of a left-tail event defined by the zp_quantile threshold.

• KLIC: Kullback-Leibler Information Criterion based density test (Corradi & Swanson, 2006). Selects the forecast whose predictive density is closest to the true density in terms of KLIC distance, evaluated via Negative Log-Likelihood Scores (NLS) under a Gaussian predictive density assumption.

• CRPS: Continuous Ranked Probability Score (Gneiting & Raftery, 2007). Jointly rewards calibration and sharpness of the predictive distribution.

• UC: Kupiec Unconditional Coverage test (Kupiec, 1995).

• MSE: Mean Squared Error.

• MAE: Mean Absolute Error.

• MASE: Mean Absolute Scaled Error.

Usage

run_comprehensive_erc_analysis(
  data_list_prepared,
  mods_matrix,
  alpha_grid,
  window_size,
  y_hat_all,
  y_raw,
  block_length = 5,
  n_boot = 999,
  zp_quantile = 0.05,
  n_crps_samples = 10,
  benchmark_col = NULL
)

Arguments

data_list_prepared	Named list of prepared data frames, one per dataset. Each element must be a list containing at least a field R_start: a non-negative integer specifying how many observations to skip from the start of y_raw before aligning with the forecast matrix. Set R_start = 0 to use all available observations. This is used as a warm-up offset when the raw series is longer than the forecast evaluation window.
mods_matrix	A legacy placeholder matrix retained for interface compatibility with external pipelines in which forecasts were previously defined as a parameter matrix. Its contents are not read or used anywhere in this function — the actual forecast structure is derived entirely from the forecast matrices supplied in y_hat_all. Pass matrix(0) when calling the function directly.
alpha_grid	Numeric scalar or vector of significance levels. Only the first element (alpha_grid[1]) is used as the significance level for all tests.
window_size	Integer window size for rolling variance estimation passed to estimate_forecast_variance.
y_hat_all	Named list of forecast results, one per dataset. Each element must be a list of length at least 3, where the third element ([[3]]) is a numeric matrix of dimension P x K_total: columns 1:(K_total-1) are the competing model forecasts and column K_total (or the column indicated by benchmark_col) is the benchmark forecast.
y_raw	Named list of raw realized value vectors, one per dataset. Each element is a numeric vector whose length must be at least R_start + P.
block_length	Integer block length for Moving Block Bootstrap (MBB) used in WRC, SPA, CPA, ZP, and KLIC tests. Default is 5. A commonly used rule of thumb is T^{1/3} (Politis & Romano, 1994); for P = 165 this gives approximately 5–6.
n_boot	integer number of MBB bootstrap replications. Default 999; see Davidson & MacKinnon (2000).
zp_quantile	Numeric quantile level used to define the left-tail threshold \tau for the ZP test, computed as quantile(realizations, zp_quantile). Default is 0.05 (5th percentile).
n_crps_samples	Integer number of Monte Carlo samples drawn from the Gaussian predictive distribution N(\hat{y}_{k,t}, \hat{\sigma}_{k,t}^2) to approximate the Continuous Ranked Probability Score (CRPS) for each forecast and time period.Default is 10 (fast, suitable for examples only). For reliable results use at least 500; for publication-quality estimates use 1000 or more. Higher values reduce Monte Carlo variance but increase computation time linearly.
benchmark_col	Index or name of the benchmark column in the forecast matrix. Defaults to the last column (NULL).

Value

list containing:

• aggregate_results: Named list of test results per dataset. Each dataset element contains named htest objects for each test and metric combination, plus VaR_Backtests (a list of per-model Kupiec htest objects).

• per_model_results: Named list of per-model Diebold-Mariano statistics per dataset, returned as data.frame objects from compute_per_model_statistics.

References

Davidson, R., & MacKinnon, J. G. (2000). Bootstrap tests: How many bootstraps? Econometric Reviews, 19(1), 55–68. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1080/07474930008800459")}

White, H. (2000). A reality check for data snooping. Econometrica, 68(5), 1097–1126. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1111/1468-0262.00152")}

Hansen, P. R. (2005). A Test for Superior Predictive Ability. Journal of Business & Economic Statistics, 23(4), 365–380. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1198/073500105000000063")}

Giacomini, R., & White, H. (2006). Tests of Conditional Predictive Ability. Econometrica, 74(6), 1545–1578. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1111/j.1468-0262.2006.00718.x")}

Corradi, V., & Swanson, N. R. (2006). Predictive density and conditional confidence interval accuracy tests. Journal of Econometrics, 135(1–2), 187–228. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1016/j.jeconom.2005.07.026")}

Corradi, V., & Swanson, N. R. (2011). The White Reality Check and some of its recent extensions. In Festschrift in honor of Halbert L. White.

Gneiting, T., & Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477), 359–378. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1198/016214506000001437")}

Kupiec, P. H. (1995). Techniques for Verifying the Accuracy of Risk Measurement Models. The Journal of Derivatives, 3(2), 173–184. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.3905/jod.1995.407942")}

Politis, D. N., & Romano, J. P. (1994). The stationary bootstrap. Journal of the American Statistical Association, 89(428), 1303–1313. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1080/01621459.1994.10476870")}

Diebold, F. X., & Mariano, R. S. (1995). Comparing Predictive Accuracy. Journal of Business & Economic Statistics, 13(3), 253–263. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1080/07350015.1995.10524599")}

See Also

create_unified_summary, generate_comprehensive_report, white_reality_check, superior_predictive_ability_test, white_reality_check_conditional, reality_check_zp_test, kullback_leibler_test, compute_kupiec

RCtest documentation built on June 2, 2026, 9:07 a.m.