reality_check_zp_test: ZP Quantile Loss Reality Check Test

In RCtest: Reality Check and Predictive Ability Tests for Forecast Evaluation

ZP Quantile Loss Reality Check Test

Description

Implements the studentized Reality Check test for comparing predictive densities based on the ZP quantile loss function of Corradi & Swanson (2006). The test evaluates whether any competing forecast more accurately predicts the probability of the outcome falling below a specified threshold than the benchmark forecast.

• H0: \max_k E[\mu^2_1(u) - \mu^2_k(u)] \leq 0 – no competing forecast has a lower expected squared probability forecast error at threshold u than the benchmark (Corradi & Swanson, 2006, eq. 7).

• H1: At least one competing forecast has lower expected ZP-loss than the benchmark.

Usage

reality_check_zp_test(
  zp_loss_differences,
  block_length,
  num_bootstrap_replications,
  alpha = 0.05
)

Arguments

zp_loss_differences	A numeric matrix (P x K) of ZP-loss differences (benchmark ZP-loss minus forecast ZP-loss), where a positive entry means the competing forecast outperforms the benchmark forecast at that period.
block_length	integer. The block length for MBB and HAC estimation. A commonly used rule of thumb is block_length \approx T^{1/3}, where T is the number of observations (Politis & Romano, 1994). For P = 165, this gives approximately 5–6.
num_bootstrap_replications	integer number of MBB bootstrap replications. Default 999; see Davidson & MacKinnon (2000).
alpha	numeric. The significance level (default 0.05).

Details

The ZP loss for forecast k at period t is

ZP_{t,k} = \left(\mathbf{1}(y_t \leq \tau) - F_k(\tau \mid \hat{y}_{t,k}, \hat{\sigma}_{t,k})\right)^2

where \tau is a threshold, F_k(\cdot) is the forecast's predictive CDF at period t, and \mathbf{1}(y_t \leq \tau) is the indicator for a tail event. Interpretively, the threshold \tau defines a tail event of interest (e.g., the 5th percentile of realizations). The ZP loss penalises the squared difference between the predicted probability of this event and whether it actually occurred. A forecast with lower ZP loss more accurately calibrates the left-tail probability. The threshold is typically set to a quantile of the realized series (e.g., quantile(realized, 0.05)); a lower threshold focuses the test more sharply on extreme left-tail events.

The benchmark is treated as a degenerate (point-mass) predictive distribution with \hat{\sigma} = 10^{-6}, which is a conservative choice ensuring the benchmark's ZP loss approximates the Brier score for the tail indicator.

Two p-values are returned: p_consistent and p_conservative, analogous to those in superior_predictive_ability_test.

Value

An object of class "htest". Lower p-values indicate that at least one competing forecast is significantly better calibrated in the left-tail than the benchmark forecast. Also contains p_consistent and p_conservative. Failure to reject H0 suggests no forecast provides significantly better density fit than the benchmark forecast.

References

Corradi, V., & Swanson, N. R. (2006). Predictive density and conditional confidence interval accuracy tests. Journal of Econometrics, 135(1–2), 187–228. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1016/j.jeconom.2005.07.026")}

Corradi, V., & Swanson, N. R. (2011). The White Reality Check and some of its recent extensions. In Festschrift in honor of Halbert L. White.

Davidson, R., & MacKinnon, J. G. (2000). Bootstrap tests: How many bootstraps? Econometric Reviews, 19(1), 55–68. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1080/07474930008800459")}

Hansen, P. R. (2005). A Test for Superior Predictive Ability. Journal of Business & Economic Statistics, 23(4), 365–380. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1198/073500105000000063")}

Politis, D. N., & Romano, J. P. (1994). The stationary bootstrap. Journal of the American Statistical Association, 89(428), 1303–1313. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1080/01621459.1994.10476870")}

Examples

data(metals)
# metals: 165 x 15; columns 1-14 are competing forecasts, column 15 is the benchmark
benchmark_col      <- 15
P                  <- nrow(metals)
K_total            <- ncol(metals)
K                  <- K_total - 1 # 14 competing forecasts
forecast_variance  <- estimate_forecast_variance(metals, benchmark_col = K_total,
                                                 window_size = 20)
comp_cols          <- setdiff(seq_len(K_total), benchmark_col)
forecast_sd_models <- sqrt(forecast_variance[, comp_cols])
threshold_val      <- quantile(metals[, K_total], 0.05)
zp_loss  <- compute_zp(metals, forecast_sd_models,
                       threshold = threshold_val, benchmark_col = K_total)
zp_diff  <- zp_loss[, K_total] - zp_loss[, comp_cols]
reality_check_zp_test(zp_diff, block_length = 5, num_bootstrap_replications = 50)

RCtest documentation built on June 2, 2026, 9:07 a.m.