PIEval: Prediction Interval (PI) evaluation

In forecTheta: Forecasting Time Series by Theta Models

Prediction Interval (PI) evaluation

Description

Implementation of Mean Scaled Interval Score (MSIS) and Absolute Coverage Difference (ACD).

Usage

PI_eval(obs, forec, lower_bounds, upper_bounds, name = c("MSIS", "ACD"), alpha = 0.05)

Arguments

obs	Observed time series values.
forec	Forecasted values corresponding to the observed data.
lower_bounds	Lower bounds of the prediction interval.
upper_bounds	Upper bounds of the prediction interval.
name	Evaluation metric to be used. Options are "MSIS" or "ACD".
alpha	Significance level for the prediction interval. Default is 0.05.

Details

The MSIS is computed by combining the width of the prediction interval with a penalty for observations falling outside the interval. This score is averaged over time and scaled by the mean absolute seasonal difference, following the same approach used for the MASE in the M4 competition, to ensure scale invariance. As a complementary metric, the ACD measures the absolute difference between the empirical and nominal coverage (e.g., 0.95), helping to assess interval reliability.

Value

Returns a numeric value corresponding to the selected evaluation metric (MSIS or ACD).

Author(s)

Jose Augusto Fiorucci, Francisco Louzada, Igor De Oliveira Barros Faluhelyi.

References

Gneiting, T. and Raftery, A.E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102 (477), 359–378, \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1198/016214506000001437")}.

Makridakis, S., Spiliotis, E. and Assimakopoulos, V. (2018). The M4 Competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting 36 (1), 54–74, \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1016/j.ijforecast.2019.04.014")}.

See Also

forecTheta-package, errorMetricFunctions, dotm, dstm, otm, stm

Examples

y1 <- 2 + 0.15 * (1:20) + rnorm(20)
y2 <- y1[20] + 0.3 * (1:30) + rnorm(30)
y <- as.ts(c(y1, y2))
out <- dotm(y, h = 10)

# MSIS metric
PI_eval(
  obs = y, forec = out$mean,
  lower_bounds = out$lower[, 3], upper_bounds = out$upper[, 3],
  name = "MSIS", alpha = 0.05
)

# ACD metric
PI_eval(
  obs = y, forec = out$mean,
  lower_bounds = out$lower[, 3], upper_bounds = out$upper[, 3],
  name = "ACD", alpha = 0.05
)

forecTheta documentation built on June 8, 2025, 10:02 a.m.