white_reality_check_cdf_approx: White's Reality Check via Expected Loss CDF Comparison...
In RCtest: Reality Check and Predictive Ability Tests for Forecast Evaluation

white_reality_check_cdf_approx

R Documentation

White's Reality Check via Expected Loss CDF Comparison (CDF-RC)

Description

Implements a studentized Reality Check test that compares competing forecasts against a benchmark across the entire distribution of loss differences, not only the mean. For each forecast k and each quantile threshold x_\tau (derived from the pooled empirical distribution of loss differences), the test evaluates whether the empirical CDF of forecast k's loss differences lies uniformly above that of the benchmark, indicating stochastic dominance of the benchmark's loss distribution over the forecast's loss distribution. Hypotheses:

H0: \max_{k,j} E[\mathbf{1}(d_{k,t} \leq x_{\tau_j})] \leq 0 for all k = 1,\ldots,K and all quantile thresholds x_{\tau_j},\ j = 1,\ldots,J – no competing forecast has a uniformly higher empirical CDF of loss differences than the benchmark at any evaluation point.
H1: At least one competing forecast has a significantly higher empirical CDF of loss differences than the benchmark at some quantile threshold x_{\tau_j}, i.e., the benchmark is stochastically dominated in terms of loss differences.

Usage

white_reality_check_cdf_approx(
  loss_differences,
  block_length,
  num_bootstrap_replications,
  alpha = 0.05
)

Arguments

`loss_differences`	A `numeric` matrix (`P x K`) of loss differences (benchmark loss minus forecast forecast loss), where `P` is the number of forecast periods and `K` is the number of competing forecasts. A positive entry means the competing forecast is more accurate than the benchmark forecast in that period.
`block_length`	`integer`. The block length for the Moving Block Bootstrap (MBB) and for HAC variance estimation via `estimate_long_run_covariance`. A commonly used rule of thumb is `T^{1/3}` (Politis & Romano, 1994). For `P = 165`, this gives approximately 5–6.
`num_bootstrap_replications`	`integer` number of MBB bootstrap replications. Default `999`; see Davidson & MacKinnon (2000).
`alpha`	`numeric`. The significance level (default `0.05`).

Details

The test proceeds in three steps.

Step 1 — Quantile grid. A grid of J = 9 evaluation points x_{\tau_1}, \ldots, x_{\tau_9} is constructed as the \tau_j \in \{0.1, 0.2, \ldots, 0.9\} quantiles of the pooled empirical distribution of all loss differences (across all forecasts and all periods). Using quantiles of the data rather than a fixed grid ensures that the evaluation points are always in the support of the observed loss differences.

Step 2 — Indicator matrix. For each forecast k and each threshold x_{\tau_j}, the binary indicator

G_{k,j,t} = \mathbf{1}(d_{k,t} \leq x_{\tau_j})

is formed, where d_{k,t} is the loss difference for forecast k at period t. This yields a P \times (K \cdot J) matrix Gdata with K \times J = 14 \times 9 = 126 columns (for the metals dataset). The column mean \bar{G}_{k,j} = \frac{1}{P}\sum_t G_{k,j,t} estimates the empirical CDF of forecast k's loss differences evaluated at x_{\tau_j}. A higher CDF value means a larger fraction of the forecast's loss differences fall below x_{\tau_j}, i.e., the competing forecast more frequently outperforms the benchmark forecast (since a positive loss difference means the competing forecast is more accurate).

Step 3 — Studentized test statistic. Each column mean is studentized by its HAC standard deviation (from estimate_long_run_covariance), and the test statistic is the maximum studentized CDF indicator mean across all K \times J columns:

\hat{T} = \max_{k,j} \frac{\bar{G}_{k,j}}{\hat{\sigma}_{k,j}}

Bootstrap p-values are obtained via the MBB of Kunsch (1989) with recentring, following the WRC procedure of White (2000) and Corradi & Swanson (2011).

Relationship to Corradi & Swanson (2006). This test is a loss-difference analogue of the predictive CDF comparison in Corradi & Swanson (2006, Section 4). Rather than comparing forecast CDFs against the true conditional distribution (as in the ZP test), it compares empirical CDFs of loss differences against zero, assessing stochastic dominance of the benchmark over each competing forecast in terms of loss. It complements white_reality_check (which tests only the mean) by detecting cases where one forecast is better in the tails but not on average.

Lower p-values are more informative: rejection of H0 indicates that at least one forecast stochastically dominates the benchmark at some point of the loss distribution. Failure to reject does not preclude dominance at specific quantiles – it means no single (k,j) combination is significant after controlling for multiple comparisons.

Value

An object of class "htest" with the following components:

`statistic`	Maximum studentized CDF indicator mean across all `K \times J` forecast-quantile combinations, labelled `"KS-type"`.
`p.value`	Bootstrap p-value from the MBB procedure.
`method`	`"Expected Loss CDF Comparison Test"`.
`null.value`	Named scalar `"max studentized CDF indicator mean (benchmark minus forecast)"` = 0.
`alternative`	Description of the alternative hypothesis.

A small p-value (below alpha) leads to rejection of H0, indicating that at least one competing forecast has a significantly higher empirical CDF of loss differences than the benchmark at some quantile threshold – i.e., the competing forecast more frequently produces smaller losses than the benchmark forecast in some region of the loss distribution.

References

Corradi, V., & Swanson, N. R. (2006). Predictive density and conditional confidence interval accuracy tests. Journal of Econometrics, 135(1–2), 187–228. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1016/j.jeconom.2005.07.026")}

Davidson, R., & MacKinnon, J. G. (2000). Bootstrap tests: How many bootstraps? Econometric Reviews, 19(1), 55–68. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1080/07474930008800459")}

White, H. (2000). A reality check for data snooping. Econometrica, 68(5), 1097–1126. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1111/1468-0262.00152")}

Corradi, V., & Swanson, N. R. (2011). The White Reality Check and some of its recent extensions. In Festschrift in honor of Halbert L. White.

Kunsch, H. R. (1989). The jackknife and the bootstrap for general stationary observations. The Annals of Statistics, 17(3), 1217–1241. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1214/aos/1176347265")}

Politis, D. N., & Romano, J. P. (1994). The stationary bootstrap. Journal of the American Statistical Association, 89(428), 1303–1313. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1080/01621459.1994.10476870")}

Examples

data(metals)
# metals: 165 x 15; columns 1-14 are competing forecasts, column 15 is the benchmark
# A small offset (+0.5) is added to the lagged benchmark to avoid degenerate zero
# loss differences when forecasts equal the realized value exactly (illustration only).
P       <- nrow(metals)
K_total <- ncol(metals)
K       <- K_total - 1L  # 14 competing forecasts
realized       <- c(metals[-1, K_total], metals[P, K_total]) + 0.5
benchmark_loss <- (metals[, K_total] - realized)^2
model_loss     <- (metals[, 1:K]     - realized)^2
loss_diff      <- benchmark_loss - model_loss
res <- white_reality_check_cdf_approx(loss_diff,
                                      block_length               = 5,
                                      num_bootstrap_replications = 50)
print(res)

RCtest documentation built on June 2, 2026, 9:07 a.m.