kullback_leibler_test: Kullback-Leibler Information Criterion (KLIC) Test

In RCtest: Reality Check and Predictive Ability Tests for Forecast Evaluation

Kullback-Leibler Information Criterion (KLIC) Test

Description

Implements the Reality Check using Negative Log-Likelihood Scores (NLS) to evaluate predictive densities in terms of their Kullback-Leibler divergence from the true density. Based on Corradi & Swanson (2006).

• H0: \max_k E[\log f_1(y_t) - \log f_k(y_t)] \leq 0 – no competing forecast achieves a higher average log-likelihood (lower KLIC distance) than the benchmark density f_1.

• H1: At least one competing forecast achieves strictly higher average log-likelihood than the benchmark.

Usage

kullback_leibler_test(
  log_likelihood_differences,
  block_length,
  num_bootstrap_replications,
  alpha = 0.05
)

Arguments

log_likelihood_differences	A numeric matrix (P x K) of Negative Log-Likelihood Score (NLS) differences: benchmark NLS minus forecast NLS. A positive entry means the forecast's density assigns higher probability to the observed outcome than the benchmark density does.
block_length	integer. The block length for MBB and HAC estimation. A commonly used rule of thumb is block_length \approx T^{1/3}, where T is the number of observations (Politis & Romano, 1994). For P = 165, this gives approximately 5–6.
num_bootstrap_replications	integer number of MBB bootstrap replications. Default 999; see Davidson & MacKinnon (2000).
alpha	numeric. The significance level (default 0.05).

Details

The KLIC between the true density f_0 and a forecast density f_k is E[\log f_0(y) - \log f_k(y)]. Minimising KLIC is equivalent to maximising expected log-likelihood. This test therefore selects the forecast with the smallest KLIC distance from the true density. Lower NLS values are better: a forecast with lower NLS assigns higher average probability to events that actually occurred (Corradi & Swanson, 2006).

The NLS loss matrix is constructed via compute_klic assuming normal predictive densities parameterised by a point forecast and a rolling-window standard deviation. The test statistic is the maximum studentized mean NLS differential, with p-values obtained via MBB following the SPA bootstrap of Hansen (2005).

Value

An object of class "htest". A small p-value (below alpha) leads to rejection of H0, indicating that at least one competing forecast has a lower Kullback-Leibler distance from the true density than the benchmark. Failure to reject H0 suggests no forecast provides significantly better density fit than the benchmark.

References

Corradi, V., & Swanson, N. R. (2006). Predictive density and conditional confidence interval accuracy tests. Journal of Econometrics, 135(1–2), 187–228. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1016/j.jeconom.2005.07.026")}

Corradi, V., & Swanson, N. R. (2011). The White Reality Check and some of its recent extensions. In Festschrift in honor of Halbert L. White.

Davidson, R., & MacKinnon, J. G. (2000). Bootstrap tests: How many bootstraps? Econometric Reviews, 19(1), 55–68. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1080/07474930008800459")}

Hansen, P. R. (2005). A Test for Superior Predictive Ability. Journal of Business & Economic Statistics, 23(4), 365–380. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1198/073500105000000063")}

Politis, D. N., & Romano, J. P. (1994). The stationary bootstrap. Journal of the American Statistical Association, 89(428), 1303–1313. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1080/01621459.1994.10476870")}

Examples

data(metals)
# metals: 165 x 15; columns 1-14 are competing forecasts, column 15 is the benchmark
benchmark_col      <- 15
P                  <- nrow(metals)
K_total            <- ncol(metals)
K                  <- K_total - 1 # 14 competing forecasts
forecast_variance  <- estimate_forecast_variance(metals, benchmark_col = K_total,
                                                 window_size = 20)
comp_cols          <- setdiff(seq_len(K_total), benchmark_col)
forecast_sd_models <- sqrt(forecast_variance[, comp_cols])
nls_loss  <- compute_klic(metals, forecast_sd_models, benchmark_col = K_total)
nls_diff  <- nls_loss[, K_total] - nls_loss[, comp_cols]
kullback_leibler_test(nls_diff, block_length = 5, num_bootstrap_replications = 50)

RCtest documentation built on June 2, 2026, 9:07 a.m.