Back Testing and Hyperparameter Tuning"
In finnts: Microsoft Finance Time Series Forecasting Framework

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

Most models automatically included in finnts, including all multivariate models, have various hyperparameters with values that need to be chosen before a model is trained. Finn solves this by leveraging the tune package within the tidymodels ecosystem.

When prep_models() is ran, hyperparameters and back test splits are calculated and written to disk. You can get the results by calling get_prepped_models().

library(finnts)

hist_data <- timetk::m4_monthly %>%
  dplyr::filter(
    date >= "2012-01-01",
    id == "M2"
  ) %>%
  dplyr::rename(Date = date) %>%
  dplyr::mutate(id = as.character(id))

run_info <- set_run_info(
  experiment_name = "finnts_fcst",
  run_name = "get_prepped_models"
)

prep_data(
  run_info = run_info,
  input_data = hist_data,
  combo_variables = c("id"),
  target_variable = "value",
  date_type = "month",
  recipes_to_run = "R1",
  forecast_horizon = 6
)

prep_models(
  run_info = run_info,
  models_to_run = c("arima", "ets", "xgboost"),
  num_hyperparameters = 10
)

model_info <- get_prepped_models(run_info = run_info)

back_test_info <- model_info %>%
  dplyr::filter(Type == "Train_Test_Splits") %>%
  dplyr::select(Data) %>%
  tidyr::unnest(Data)

print(back_test_info)
print(unique(back_test_info$Run_Type))

model_workflow_info <- model_info %>%
  dplyr::filter(Type == "Model_Workflows") %>%
  dplyr::select(Data) %>%
  tidyr::unnest(Data)

print(model_workflow_info)

model_hyperparameter_info <- model_info %>%
  dplyr::filter(Type == "Model_Hyperparameters") %>%
  dplyr::select(Data) %>%
  tidyr::unnest(Data)

print(model_hyperparameter_info)

The above outputs allow a Finn user to understand what hyperparameters are chosen for tuning and how the model refitting process will work. When tuning hyperparameters, Finn uses the "Validation" train/test splits, with the final parameters chosen using RMSE. For some models like ARIMA that don't follow a traditional hyperparameter tuning process, the model is fit from scratch across all train/test splits. After hyperparameters are chosen, the model is refit across the "Back_Test" and "Future_Forecast" splits. The "Back_Test" splits are the true testing data that will be used when selecting the final "Best-Model". "Ensemble" splits are also created as ensemble training data if ensemble models are chosen to run. Ensemble models follow a similar tuning process.