generate_comprehensive_report: Generate Comprehensive Markdown Report

In RCtest: Reality Check and Predictive Ability Tests for Forecast Evaluation

Generate Comprehensive Markdown Report

Description

Generates an automatic summary report in Markdown format covering all error metrics (MSE, MAE, MASE) and distributional tests (ZP, KLIC, Kupiec). For the ZP and KLIC sections, the report lists superior competing forecasts (i.e. those whose forecasts are found to be more accurate than the benchmark forecast) or states that no superior forecasts were found. For the Kupiec section, forecasts with correct VaR coverage (Reject_H0 == FALSE) are listed, or a message is printed if none passed.

Technical Abbreviations:

• WRC: White's Reality Check (White, 2000). Tests whether any competing forecast has lower expected loss than the benchmark forecast; controls family-wise error rate.

• SPA: Superior Predictive Ability test (Hansen, 2005). A studentized extension of WRC with improved power that corrects for irrelevant forecasts.

• CPA: Conditional Predictive Ability test (Giacomini & White, 2006). Tests whether loss differentials are predictable by a conditioning variable.

• ZP: Quantile Loss test (Corradi & Swanson, 2006). Evaluates whether any competing forecast better calibrates the probability of a left-tail event defined by the zp_quantile threshold.

• KLIC: Kullback-Leibler Information Criterion based density test (Corradi & Swanson, 2006). Selects the forecast whose predictive density is closest to the true density in terms of KLIC distance, evaluated via Negative Log-Likelihood Scores (NLS) under a Gaussian predictive density assumption.

• CRPS: Continuous Ranked Probability Score (Gneiting & Raftery, 2007). Jointly rewards calibration and sharpness of the predictive distribution.

• UC: Kupiec Unconditional Coverage test (Kupiec, 1995).

• MSE: Mean Squared Error.

• MAE: Mean Absolute Error.

• MASE: Mean Absolute Scaled Error.

Usage

generate_comprehensive_report(
  summary_df,
  zp_models_df,
  klic_models_df,
  kupiec_models_df,
  dataset_name,
  alpha = 0.05
)

Arguments

summary_df	data.frame unified summary from create_unified_summary.
zp_models_df	data.frame with columns Model, P_Value, and Dataset. Models with P_Value <= alpha are considered superior and listed in the report; if none are found, a message \"No superior models found\" is printed. Pass all competing models to ensure complete and unbiased reporting.
klic_models_df	data.frame with columns Model, P_Value, and Dataset. Models with P_Value <= alpha are considered superior and listed in the report; if none are found, a message \"No superior models found\" is printed. Pass all competing models to ensure complete and unbiased reporting.
kupiec_models_df	data.frame with columns Model, Reject_H0, and Dataset. Contains all models tested under the Kupiec UC test. Models with Reject_H0 == FALSE are considered to have correct VaR coverage and are listed in the report; models with Reject_H0 == TRUE are excluded. Pass all competing models to ensure complete and unbiased reporting.
dataset_name	character the name of the dataset to be used in the report header.
alpha	numeric significance level (default 0.05).

Value

character string in Markdown format.

References

White, H. (2000). A reality check for data snooping. Econometrica, 68(5), 1097–1126. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1111/1468-0262.00152")}

Hansen, P. R. (2005). A Test for Superior Predictive Ability. Journal of Business & Economic Statistics, 23(4), 365–380. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1198/073500105000000063")}

Giacomini, R., & White, H. (2006). Tests of Conditional Predictive Ability. Econometrica, 74(6), 1545–1578. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1111/j.1468-0262.2006.00718.x")}

Corradi, V., & Swanson, N. R. (2006). Predictive density and conditional confidence interval accuracy tests. Journal of Econometrics, 135(1–2), 187–228. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1016/j.jeconom.2005.07.026")}

Corradi, V., & Swanson, N. R. (2011). The White Reality Check and some of its recent extensions. In Festschrift in honor of Halbert L. White.

Gneiting, T., & Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477), 359–378. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1198/016214506000001437")}

Kupiec, P. H. (1995). Techniques for Verifying the Accuracy of Risk Measurement Models. The Journal of Derivatives, 3(2), 173–184. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.3905/jod.1995.407942")}

Politis, D. N., & Romano, J. P. (1994). The stationary bootstrap. Journal of the American Statistical Association, 89(428), 1303–1313. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1080/01621459.1994.10476870")}

Diebold, F. X., & Mariano, R. S. (1995). Comparing Predictive Accuracy. Journal of Business & Economic Statistics, 13(3), 253–263. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1080/07350015.1995.10524599")}

Examples

data(metals)
ds_name       <- "Dataset1"
prep_list     <- list(Dataset1 = list(R_start = 0))
realizations  <- list(Dataset1 = metals[, ncol(metals)])
f_hat         <- list(list(NULL, NULL, metals))
names(f_hat)  <- ds_name
res <- run_comprehensive_erc_analysis(
  data_list_prepared = prep_list,
  mods_matrix        = matrix(0),
  alpha_grid         = 0.05,
  window_size        = 20,
  y_hat_all          = f_hat,
  y_raw              = realizations,
  block_length       = 5,
  n_boot             = 10,
  zp_quantile        = 0.05
)
unified_summ <- create_unified_summary(res$aggregate_results)

zp_table     <- data.frame(Model   = colnames(metals)[1:(ncol(metals) - 1)],
                            P_Value = 0.1, Dataset = ds_name)
klic_table   <- zp_table
kupiec_table <- data.frame(Model     = colnames(metals)[1:(ncol(metals) - 1)],
                            Reject_H0 = rep(FALSE, ncol(metals) - 1),
                            Dataset   = ds_name)

report <- generate_comprehensive_report(
  summary_df       = unified_summ$summary,
  zp_models_df     = zp_table,
  klic_models_df   = klic_table,
  kupiec_models_df = kupiec_table,
  dataset_name     = ds_name
)
cat(report)

RCtest documentation built on June 2, 2026, 9:07 a.m.