Introduction to 'ForecastTB' package"

  collapse = TRUE,
  comment = "#>"

Demonstration of 'ForecastTB' package:

This document demonstates the R package 'ForecastTB'. It is intended for comparing the performance of forecasting methods. The package assists in developing background, strategies, policies and environment needed for comparison of forecasting methods. A comparison report for the defined framework is produced as an output. Load the package as following:


The basic function of the package is prediction_errors(). Following are the parameters considered by this function:

The prediction_errors() function returns, two slots as output. First slot is output, which provides Error_Parameters, indicating error values for the forecasting methods and error parameters defined in the framework, and Predicted_Values as values forecasted with the same foreasting methods. Further, the second slot is parameters, which returns the parameters used or provided to prediction_errors() function.

a <- prediction_errors(data = nottem)  #`nottem` is a sample dataset in CRAN


The quick visualization of the object retuned with prediction_errors() function can be done with plot() function as below:

b <- plot(a)

Comparison of multiple methods:

As discussed above, prediction_errors() function evaluates the performance of ARIMA method. In addition, it allows to compare performance of distinct methods along with ARIMA. In following example, two methods (LPSF and PSF) are compared along with the ARIMA. These methods are formatted in the form of a function, which requires data and nval as input parameters and must return the nval number of frecasted values as a vector. In following code, test1() and test2() functions are used for LPSF and PSF methods, respectively.

test1 <- function(data, nval){
   return(lpsf(data = data, n.ahead = nval))

test2 <- function(data, nval){
  a <- psf(data = data, cycle = 12)
  b <- predict(object = a, n.ahead = nval)

Following code chunk show how user can attach various methods in the prediction_errors() function. In this chunk, the append_ parameter is assigned 1, to appned the new methods (LPSF and PSF) in addition to the default ARIMA method. On contrary, if the append_parameter is assigned 0, only newly added LPSF and PSF nethods would be compared.

a1 <- prediction_errors(data = nottem, nval = 48, 
                        Method = c("test1(data, nval)", "test2(data, nval)"), 
                        MethodName = c("LPSF","PSF"), append_ = 1)
b1 <- plot(a1)

Appending new methods:

Consider, another function test3(), which is to be added to an already existing object prediction_errors, eg. a1.

test3 <- function(data, nval){
  b <- as.numeric(forecast(ets(data), h = nval)$mean)

For this purpose, the append_() function can be used as follows:

The append_() function have object, Method, MethodName, ePara and ePara_name parameters, with similar meaning as that of used in prediction_errors() function. Other hidden parameters of the append_() function automatically get synced with the prediction_errors() function.

c1 <- append_(object = a1, Method = c("test3(data,nval)"), MethodName = c('ETS'))
d1 <- plot(c1)

Removing methods:

When more than one methods are established in the environment and the user wish to remove one or more of these methods from it, the choose_() function can be used. This function takes a prediction_errors object as input shows all methods established in the environment, and asks the number of methods which the user wants to remove from it.

In the following example, the user supplied 4 as input, which reflects Method 4: ETS, and in response to this, the choose_() function provides a new object with updated method lists.

# > e1 <- choose_(object = c1)
# Following are the methods attached with the object:
#         [,1]    [,2]   [,3]  [,4] 
# Indices "1"     "2"    "3"   "4"  
# Methods "ARIMA" "LPSF" "PSF" "ETS"
# Enter the indices of methods to remove:4
# > e1@output$Error_Parameters
#            RMSE       MAE exec_time
# ARIMA 2.5233156 2.1280641 0.1963789
# LPSF  2.3915796 1.9361111 0.2990961
# PSF   2.2748736 1.8301389 0.1226711

Adding new Error metrics:

In default scenario, the prediction_errors() function compares forecasting methods in terms of RMSE, MAE and MAPE. In addition, it allows to append multiple new error metrics. The Percent change in variance (PCV) is an another error metric with following definition:

$PCV = \frac{\mid var(Predicted) - var(Observed) \mid}{var(Observed)}$

where $var(Predicted)$ and $var(Observed)$ are variance of predicted and obvserved values. Following chunk code is the function for PCV error metric:

pcv <- function(obs, pred){
  d <- (var(obs) - var(pred)) * 100/ var(obs)
  d <- abs(as.numeric(d))

Following chunk code is used to append PCV as a new error metric in existing prediction_errors object.

a1 <- prediction_errors(data = nottem, nval = 48, 
                        Method = c("test1(data, nval)", "test2(data, nval)"), 
                        MethodName = c("LPSF","PSF"), 
                        ePara = "pcv(obs, pred)", ePara_name = 'PCV',
                        append_ = 1)
b1 <- plot(a1)

A unique plot:

A unique way of showing forecasted values, especially if these are seasonal values, the following function can be used. This plot shows how forecatsed observations are behaving on an increasing number of seasonal time horizons.


Monte-Carlo strategy:

Monte-Carlo is a popular strategy to compare the performance of forecasting methods, which selects multiple patches of dataset randomly and test performance of forecasting methods and returns the average error values.

The Monte-Carlo strategy ensures an accurate comparison of forecasting methods and avoids the baised results obtained by chance.

This package provides the monte_carlo() function as follows:

The parameters used in this function are:

This function returns:

a1 <- prediction_errors(data = nottem, nval = 48, 
                        Method = c("test1(data, nval)"), 
                        MethodName = c("LPSF"), append_ = 1)
monte_carlo(object = a1, size = 180, iteration = 10)

When monte_carlo() function with fval and figs ON flags:

monte_carlo(object = a1, size = 144, iteration = 2, fval = 1, figs = 1)

Functions in Future Versions:

Try the ForecastTB package in your browser

Any scripts or data that you put into this service are public.

ForecastTB documentation built on March 14, 2020, 5:07 p.m.