metrics: Forecast Performance Metrics

In tsaux: Time Series Forecasting Auxiliary Functions

Forecast Performance Metrics

Description

Functions to calculate a number of performance metrics.

Usage

mape(actual, predicted)

bias(actual, predicted)

mslre(actual, predicted)

mase(actual, predicted, original_series = NULL, frequency = 1)

mis(actual, lower, upper, alpha)

wape(actual, predicted, weights)

wslre(actual, predicted, weights)

wse(actual, predicted, weights)

pinball(actual, distribution, alpha = 0.1)

crps(actual, distribution)

rmape(actual, predicted)

smape(actual, predicted)

msis(actual, lower, upper, original_series, frequency = 1, alpha)

Arguments

actual	the actual values corresponding to the forecast period.
predicted	the predicted values corresponding to the forecast period.
original_series	the actual values corresponding to the training period.
frequency	the seasonal frequency of the series used in the model.
lower	the lower distributional forecast for the quantile corresponding to the coverage ratio alpha (i.e. alpha/2).
upper	the upper distributional forecast for the quantile corresponding to the coverage ratio alpha (i.e. 1 - alpha/2).
alpha	the distributional coverage.
weights	a vector of weights for generating weighted metrics. If the actual and predicted inputs are univariate, this should be equal to the length of the actual series and calculates a time-weighted average; otherwise, the weights should be of length equal to the number of series in a multivariate case, in which case a cross-sectional average is calculated.
distribution	the forecast distribution (returned in the distribution slot of the prediction object). This is used in the continuous ranked probability score (crps) of Gneiting et al. (2005), and calculated using the function from the 'scoringRules' package.

Details

The following performance metrics are implemented:

Mean Absolute Percentage Error (MAPE)

Measures the average percentage deviation of predictions from actual values.

MAPE = \frac{1}{n} \sum_{t=1}^{n} \left| \frac{A_t - P_t}{A_t} \right|

where A_t is the actual value and P_t is the predicted value.

Rescaled Mean Absolute Percentage Error (RMAPE)

A transformation of MAPE using a Box-Cox transformation for scale invariance (Swanson et al.).

Symmetric Mean Absolute Percentage Error (SMAPE)

An alternative to MAPE that symmetrizes the denominator.

SMAPE = \frac{2}{n} \sum_{t=1}^{n} \frac{|A_t - P_t|}{|A_t| + |P_t|}

Mean Absolute Scaled Error (MASE)

Compares the absolute error to the mean absolute error of a naive seasonal forecast.

MASE = \frac{\frac{1}{n} \sum_{t=1}^{n} |P_t - A_t|}{\frac{1}{N-s} \sum_{t=s+1}^{N} |A_t - A_{t-s}|}

where s is the seasonal period.

Mean Squared Logarithmic Relative Error (MSLRE)

Measures squared log relative errors to penalize large deviations.

MSLRE = \frac{1}{n} \sum_{t=1}^{n} \left( \log(1 + A_t) - \log(1 + P_t) \right)^2

Mean Interval Score (MIS)

Evaluates the accuracy of prediction intervals.

MIS = \frac{1}{n} \sum_{t=1}^{n} (U_t - L_t) + \frac{2}{\alpha} [(L_t - A_t) I(A_t < L_t) + (A_t - U_t) I(A_t > U_t)]

where L_t and U_t are the lower and upper bounds of the interval.

Mean Scaled Interval Score (MSIS)

A scaled version of MIS, dividing by the mean absolute seasonal error.

MSIS = \frac{1}{h} \sum_{t=1}^{h} \frac{(U_t - L_t) + \frac{2}{\alpha} [(L_t - A_t) I(A_t < L_t) + (A_t - U_t) I(A_t > U_t)]}{\frac{1}{N-s} \sum_{t=s+1}^{N} |A_t - A_{t-s}|}

Bias

Measures systematic overestimation or underestimation.

Bias = \frac{1}{n} \sum_{t=1}^{n} (P_t - A_t)

Weighted Absolute Percentage Error (WAPE)

A weighted version of MAPE.

WAPE = \sum_{t=1}^{n} \mathbf{w} \frac{|P_t - A_t|}{A_t}

where \mathbf{w} is the weight vector.

Weighted Squared Logarithmic Relative Error (WSLRE)

A weighted version of squared log relative errors.

WSLRE = \sum_{t=1}^{n} \mathbf{w} (\log(P_t / A_t))^2

Weighted Squared Error (WSE)

A weighted version of squared errors.

WSE = \sum_{t=1}^{n} \mathbf{w} \left( \frac{P_t}{A_t} \right)^2

Pinball Loss

A scoring rule used for quantile forecasts.

\text{Pinball} = \frac{1}{n} \sum_{t=1}^{n} \left[ \tau (A_t - Q^\tau_t) I(A_t \geq Q^\tau_t) + (1 - \tau) (Q^\tau_t - A_t) I(A_t < Q^\tau_t) \right]

where

Q^\tau_t

is the predicted quantile at level

\tau

.

Continuous Ranked Probability Score (CRPS)

A measure of probabilistic forecast accuracy.

CRPS = \frac{1}{n} \sum_{t=1}^{n} \int_{-\infty}^{\infty} (F_t(y) - I(y \geq A_t))^2 dy

where F_t(y) is the cumulative forecast distribution.

Value

A numeric value.

Note

The RMAPE is the rescaled measure for MAPE based on the paper by Swanson et al.

Author(s)

Alexios Galanos

References

\insertRef

Tofallis2015tsaux
\insertRefHyndman2006tsaux
\insertRefGneiting2005tsaux
\insertRefGneiting2007tsaux
\insertRefSwanson2011tsaux

tsaux documentation built on April 4, 2025, 3:08 a.m.