score.mvgam_forecast: Compute probabilistic forecast scores for 'mvgam' models

In mvgam: Multivariate (Dynamic) Generalized Additive Models

Compute probabilistic forecast scores for mvgam models

Description

Compute probabilistic forecast scores for mvgam models

Usage

## S3 method for class 'mvgam_forecast'
score(
  object,
  score = "crps",
  log = FALSE,
  weights,
  interval_width = 0.9,
  n_cores = 1,
  ...
)

score(object, ...)

Arguments

object	mvgam_forecast object. See forecast.mvgam().
score	character specifying the type of proper scoring rule to use for evaluation. Options are: sis (i.e. the Scaled Interval Score), energy, variogram, elpd (i.e. the Expected log pointwise Predictive Density), drps (i.e. the Discrete Rank Probability Score), crps (the Continuous Rank Probability Score) or brier (the latter of which is only applicable for bernoulli models. Note that when choosing elpd, the supplied object must have forecasts on the link scale so that expectations can be calculated prior to scoring. If choosing brier, the object must have forecasts on the expected scale (i.e. probability predictions). For all other scores, forecasts should be supplied on the response scale (i.e. posterior predictions)
log	logical. Should the forecasts and truths be logged prior to scoring? This is often appropriate for comparing performance of models when series vary in their observation ranges. Ignored if score = 'brier'
weights	optional vector of weights (where length(weights) == n_series) for weighting pairwise correlations when evaluating the variogram score for multivariate forecasts. Useful for down-weighting series that have larger magnitude observations or that are of less interest when forecasting. Ignored if score != 'variogram'
interval_width	proportional value on ⁠[0.05,0.95]⁠ defining the forecast interval for calculating coverage and, if score = 'sis', for calculating the interval score. Ignored if score = 'brier'
n_cores	integer specifying number of cores for calculating scores in parallel
...	Ignored

Value

A list containing scores and interval coverages per forecast horizon. If score %in% c('drps', 'crps', 'elpd', 'brier'), the list will also contain return the sum of all series-level scores per horizon. If score %in% c('energy','variogram'), no series-level scores are computed and the only score returned will be for all series. For all scores apart from elpd and brier, the in_interval column in each series-level slot is a binary indicator of whether or not the true value was within the forecast's corresponding posterior empirical quantiles. Intervals are not calculated when using elpd because forecasts will only contain the linear predictors

Author(s)

Nicholas J Clark

References

Gneiting, T. and Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477), 359-378. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1198/016214506000001437")}

See Also

forecast.mvgam, ensemble

Examples

## Not run: 
# Simulate observations for three count-valued time series
data <- sim_mvgam()

# Fit a dynamic model using 'newdata' to automatically produce forecasts
mod <- mvgam(
  y ~ 1,
  trend_model = RW(),
  data = data$data_train,
  newdata = data$data_test,
  chains = 2,
  silent = 2
)

# Extract forecasts into a 'mvgam_forecast' object
fc <- forecast(mod)
plot(fc)

# Compute Discrete Rank Probability Scores and 0.90 interval coverages
fc_scores <- score(fc, score = 'drps')
str(fc_scores)

# An example using binary data
data <- sim_mvgam(family = bernoulli())

mod <- mvgam(
  y ~ s(season, bs = 'cc', k = 6),
  trend_model = AR(),
  data = data$data_train,
  newdata = data$data_test,
  family = bernoulli(),
  chains = 2,
  silent = 2
)

# Extract forecasts on the expectation (probability) scale
fc <- forecast(mod, type = 'expected')
plot(fc)

# Compute Brier scores
fc_scores <- score(fc, score = 'brier')
str(fc_scores)

## End(Not run)

mvgam documentation built on Jan. 21, 2026, 9:07 a.m.