compute_per_model_statistics: Per-Model Diebold-Mariano Test (HAC + MBB Bootstrap)

In RCtest: Reality Check and Predictive Ability Tests for Forecast Evaluation

Per-Model Diebold-Mariano Test (HAC + MBB Bootstrap)

Description

Performs a Diebold-Mariano (1995) test for each competing forecast separately, testing whether the predictive accuracy of that forecast differs significantly from the benchmark forecast. The test statistic is the sample mean loss differential standardised by a Newey-West HAC standard error; p-values are provided both analytically and via Moving Block Bootstrap (MBB). The direction of the alternative hypothesis is controlled by the H1 argument: "same" for a two-sided test (H_1: \bar{d}_k \neq 0), "more" for the one-sided test that forecast k is more accurate than the benchmark (H_1: \bar{d}_k > 0), and "less" for the one-sided test that forecast k is less accurate than the benchmark (H_1: \bar{d}_k < 0).

Usage

compute_per_model_statistics(
  loss_differences,
  model_names,
  n_boot = 999,
  block_length = 5,
  alpha = 0.05,
  h = 1,
  H1 = "same"
)

Arguments

loss_differences	matrix of dimension P x K, where P is the number of forecast periods and K is the number of competing forecasts. Each column k contains the loss differential series d_{t,k} = g(e_{0,t}) - g(e_{k,t}) for a generic loss function g. In the standard workflow of this package (run_comprehensive_erc_analysis), g is the squared error loss, so d_{t,k} = (y_t - \hat{y}_{t,0})^2 - (y_t - \hat{y}_{t,k})^2. A positive value of d_{t,k} means the forecast from model k is more accurate than the benchmark forecast in period t.
model_names	character vector of length K with names of the competing model forecasts.
n_boot	integer number of MBB replications for P_Value_Boot. Default 999; see Davidson & MacKinnon (2000).
block_length	integer block length for HAC and MBB. Rule of thumb: T^{1/3}; for P = 165 approximately 5–6. Default is 5.
alpha	numeric significance level. Default 0.05.
h	integer forecast horizon (number of steps ahead). Default is 1 (one-step-ahead). Passed to the Harvey, Leybourne & Newbold (1997) small-sample correction; see Details.
H1	character alternative hypothesis: "same" (two-sided, default), "more" (one-sided, forecast k better), or "less" (one-sided, forecast k worse). See Details.

Details

For each forecast k, the loss differential series is:

d_{t,k} = g(e_{0,t}) - g(e_{k,t})

where g(\cdot) is the loss function used to construct loss_differences. In the standard workflow of this package (run_comprehensive_erc_analysis), g is the squared error loss:

d_{t,k} = (y_t - \hat{y}_{t,0})^2 - (y_t - \hat{y}_{t,k})^2

The Diebold-Mariano test statistic is:

DM_k = \frac{\bar{d}_k}{\hat{SE}_{HAC,k}}

For multi-step forecasts (h > 1), the Harvey, Leybourne & Newbold (1997) small-sample correction is applied:

DM^*_k = DM_k \times \sqrt{\frac{T + 1 - 2h + \frac{1}{T}h(h-1)}{T}}

For h = 1 the correction reduces to \sqrt{(T-1)/T}, which approaches 1 as T \to \infty. For h > 1 the correction inflates the statistic, improving finite-sample size control. The corrected statistic DM^*_k is compared to a t(T-1) distribution. The analytic p-value (P_Value) uses the t-distribution with P - 1 degrees of freedom. The bootstrap p-value (P_Value_Boot) uses MBB resampling (Kunsch, 1989) with recentering at the sample mean \bar{d}_k, placing the bootstrap distribution under H0. The Harvey correction is applied consistently to both the analytic and bootstrap statistics. The p-values are computed according to the alternative hypothesis specified by H1:

Alternative Hypothesis and P-values (H1)

"same" – two-sided

p = 2 \cdot P(t_{T-1} < -|DM^*_k|)

"more" – one-sided right

p = P(t_{T-1} > DM^*_k)

"less" – one-sided left

p = P(t_{T-1} < DM^*_k)

The bootstrap analogue replaces the t-distribution probability with the empirical proportion of bootstrap statistics falling in the appropriate tail.

where T follows a t-distribution with P - 1 degrees of freedom. The bootstrap analogue replaces the t-distribution tail probability with the empirical proportion of bootstrap statistics falling in the appropriate tail.

This function performs K individual tests and does not control for multiple comparisons. For a joint test controlling the family-wise error rate, use white_reality_check or superior_predictive_ability_test.

Note on MASE: When loss_differences are constructed from Mean Absolute Scaled Errors, the scaling (division by the naive benchmark MAE, i.e. mean(abs(diff(realizations)))) must be applied before passing the loss differentials to this function. compute_per_model_statistics receives pre-computed loss differentials and applies no internal rescaling — the caller is responsible for ensuring that MASE-based loss_differences already contain scaled errors. In run_comprehensive_erc_analysis this is handled automatically.

Value

data.frame with one row per competing model forecast:

Model	Model name.
Mean_Loss_Diff	Sample mean of d_{t,k}.
Frac_Better_Than_Benchmark	Fraction of periods where d_{t,k} > 0.
T_Stat	Harvey-corrected DM statistic DM^*_k.
P_Value	Analytic p-value (t-distribution, T-1 df).
P_Value_Boot	MBB bootstrap p-value.
Significant	TRUE if P_Value <= alpha.
Significant_Boot	TRUE if P_Value_Boot <= alpha.

References

Davidson, R., & MacKinnon, J. G. (2000). Bootstrap tests: How many bootstraps? Econometric Reviews, 19(1), 55–68. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1080/07474930008800459")}

Diebold, F. X., & Mariano, R. S. (1995). Comparing Predictive Accuracy. Journal of Business & Economic Statistics, 13(3), 253–263. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1080/07350015.1995.10524599")}

Harvey, D., Leybourne, S., & Newbold, P. (1997). Testing the equality of prediction mean squared errors. International Journal of Forecasting, 13(2), 281–291. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1016/S0169-2070(96)00719-4")}

Kunsch, H. R. (1989). The jackknife and the bootstrap for general stationary observations. The Annals of Statistics, 17(3), 1217–1241. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1214/aos/1176347265")}

Newey, W. K., & West, K. D. (1987). A Simple, Positive Semi-Definite, Heteroskedasticity and Autocorrelation Consistent Covariance Matrix. Econometrica, 55(3), 703–708. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.2307/1913610")}

See Also

white_reality_check, superior_predictive_ability_test, estimate_long_run_covariance

Examples

data(metals)
P       <- nrow(metals)
K_total <- ncol(metals)
K       <- K_total - 1
# A small offset (+0.5) avoids degenerate zero loss differences (illustration only).
realized         <- c(metals[-1, K_total], metals[P, K_total]) + 0.5
bench_loss       <- (metals[, K_total] - realized)^2
model_loss       <- (metals[, 1:K]     - realized)^2
loss_differences <- bench_loss - model_loss
model_names      <- colnames(metals)[1:K]
# Two-sided test (default)
result_df <- compute_per_model_statistics(loss_differences, model_names,
                                          n_boot = 10)
print(result_df)
# One-sided test: H1 = forecast is more accurate than benchmark
result_more <- compute_per_model_statistics(loss_differences, model_names,
                                            n_boot = 10, H1 = "more")
print(result_more)

RCtest documentation built on June 2, 2026, 9:07 a.m.