score_forecasts: Score forecasts

In reichlab/covidHubUtils: Utility functions for the COVID-19 forecast hub

Score forecasts

Description

Score forecasts

Usage

score_forecasts(
  forecasts,
  truth,
  return_format = "wide",
  metrics = c("abs_error", "wis", "wis_components", "interval_coverage",
    "quantile_coverage"),
  use_median_as_point = FALSE
)

Arguments

forecasts	required data.frame with forecasts in the format returned by load_forecasts
truth	required data.frame with forecasts in the format returned by load_truth
return_format	string: "long" returns long format with a column for "score_name" and a column for "score_value"; "wide" returns wide format with a separate column for each score. Defaults to "wide".
metrics	character vector of the metrics to be returned with options "abs_error", "wis", "wis_components","interval_coverage", and "quantile_coverage"
use_median_as_point	logical: TRUE uses the median as the point forecast when scoring; FALSE uses the point forecasts from the data when scoring. Defaults to FALSE

Value

data.frame with scores. The result will have some columns that define the observation, namely, model, forecast_date, location, horizon, temporal_resolution, target_variable, horizon, and target_end_date. Other columns will contain scores dependent on metrics parameter:

• true_value is the observed truth at that location and target_end_date (always returned)

• abs_error is the absolute error based on median estimate if use_median_as_point is TRUE, and absolute error based on point forecast if use_median_as_point is FALSE

• wis is the weighted interval score

• dispersion the component of WIS made up of interval widths

• overprediction the component of WIS made up of overprediction of intervals

• underprediction the component of WIS made up of underprediction of intervals

• coverage_X are prediction interval coverage at alpha level X

• quantile_coverage_0.X are one-sided quantile coverage at quantile X

If return_format is "long", also contains columns score_name and score_value where score_name is the type of score calculated and score_value has the numeric value of the score. If return_format is "wide", each calculated score is in its own column.

References

Bracher J, Ray EL, Gneiting T, Reich NG. (2020) Evaluating epidemic forecasts in an interval format. arXiv:2005.12881. https://arxiv.org/abs/2005.12881.

Examples

library(scoringutils)
forecasts <- load_latest_forecasts(
  models = c("COVIDhub-ensemble", "UMass-MechBayes"),
  last_forecast_date = "2020-12-14",
  forecast_date_window_size = 7,
  locations = c("US"),
  targets = paste(1:4, "wk ahead inc death"),
  source = "zoltar"
)
truth <- load_truth("JHU", target_variable = "inc death", locations = "US")
score_forecasts(forecasts, truth)

forecasts <- load_latest_forecasts(
  models = c("ILM-EKF"),
  hub = c("ECDC", "US"), last_forecast_date = "2021-03-08",
  forecast_date_window_size = 0,
  locations = c("GB"),
  targets = paste(1:4, "wk ahead inc death"),
  source = "zoltar"
)
truth <- load_truth("JHU",
  hub = c("ECDC", "US"),
  target_variable = "inc death", locations = "GB"
)
score_forecasts(forecasts, truth)

reichlab/covidHubUtils documentation built on Feb. 6, 2024, 1:42 p.m.