In epiforecasts/scoringutils: Utilities for Scoring and Assessing Predictions

knitr::opts_chunk$set(echo = TRUE,
                      fig.width = 7,
                      collapse = TRUE,
                      comment = "#>",
                      fig.path = "man/figures/")
library(scoringutils)
library(magrittr)
library(data.table)
library(ggplot2)
library(knitr)

## code to determine version inspired by [pkgdown:::dev_mode_auto()]
version <- packageVersion("scoringutils")
devel <- length(unclass(package_version(version))[[1]]) > 3

if (devel) {
  cat(
    "**Note**: ",
    "[This documentation](https://epiforecasts.io/scoringutils/dev) refers to the development version of `scoringutils`. ",
    "You can also view the [documentation of the stable version]",
    "(https://epiforecasts.io/scoringutils).",
    sep = ""
  )
} else {
  cat(
    "**Note**: ",
    "[This documentation](https://epiforecasts.io/scoringutils) refers to the stable version of `scoringutils`. ",
    "You can also view the [documentation of the development version]",
    "(https://epiforecasts.io/scoringutils/dev).",
    sep = ""
  )
}
cat("\n\n")

The scoringutils package facilitates the process of evaluating forecasts in R, using a convenient and flexible data.table-based framework. It provides broad functionality to check the input data and diagnose issues, to visualise forecasts and missing data, to transform data before scoring, to handle missing forecasts, to aggregate scores, and to visualise the results of the evaluation. The package is easily extendable, meaning that users can supply their own scoring rules or extend existing classes to handle new types of forecasts.

The package underwent a major re-write. The most comprehensive documentation for the updated package is the revised version of our original scoringutils paper.

Another good starting point are the vignettes on Getting started, Details on the metrics implemented and Scoring forecasts directly.

For further details on the specific issue of transforming forecasts for scoring see:

Nikos I. Bosse, Sam Abbott, Anne Cori, Edwin van Leeuwen, Johannes Bracher* and Sebastian Funk* (*: equal contribution) (2023). Scoring epidemiological forecasts on transformed scales, PLoS Comput Biol 19(8): e1011393 https://doi.org/10.1371/journal.pcbi.1011393

Installation

Install the CRAN version of this package using

install.packages("scoringutils")

Install the unstable development version from GitHub using

remotes::install_github("epiforecasts/scoringutils", dependencies = TRUE)

Quick start

Forecast types

scoringutils currently supports scoring the following forecast types: - binary: a probability for a binary (yes/no) outcome variable. - point: a forecast for a continuous or discrete outcome variable that is represented by a single number. - quantile: a probabilistic forecast for a continuous or discrete outcome variable, with the forecast distribution represented by a set of predictive quantiles. - sample: a probabilistic forecast for a continuous or discrete outcome variable, with the forecast represented by a finite set of samples drawn from the predictive distribution.

Input formats and input validation

The expected input format is generally a data.frame (or similar) with required columns observed, predicted, and model that holds the forecasts and observed values. Exact requirements depend on the forecast type. For more information, have a look at the paper, call ?as_forecast(), or have a look at the example data provided in the package (example_binary, example_point, example_quantile, example_sample_continuous, example_sample_discrete).

Before scoring, input data needs to be validated and transformed into a forecast object using the function as_forecast().

forecast_quantile <- example_quantile |>
  as_forecast(
    forecast_unit = c(
      "location", "forecast_date", "target_end_date", "target_type", "model", "horizon"
    ),
    forecast_type = "quantile"
  )

print(forecast_quantile, 2)

The forecast unit

For quantile-based and sample-based forecasts, a single prediction is represented by a set of several quantiles (or samples) from the predictive distribution, i.e. several rows in the input data. scoringutils therefore needs to group rows together that form a single forecast. scoringutils uses all other existing columns in the input data to achieve this - the values in all other columns should uniquely identify a single forecast. Additional columns unrelated to the forecast unit can mess this up. The forecast_unit argument in as_forecast() makes sure that only those columns are retained which are relevant for defining the unit of a single forecast.

Scoring forecasts

Forecasts can be scored by calling score() on a validated forecast object.

scores <- forecast_quantile |> 
  score()

score() takes an additional argument, metrics, with a list of scoring rules. Every forecast type has a default list of metrics. You can easily add your own scoring functions, as long as they conform with the format for that forecast type. See the paper for more information.

You can summarise scores using the function summarise_scores(). The by argument is used to specify the desired level of summary. fun let's you specify any summary function, although it is recommended to stick to the mean as a primary summary function, as other functions can lead to improper scores.

scores |> 
  summarise_scores(by = c("model", "target_type")) |>
  summarise_scores(by = c("model", "target_type"), fun = signif, digits = 3)

Package workflow

The following depicts the suggested workflow for evaluating forecasts with scoringutils (sections refer to the paper). Please find more information in the paper, the function documentation and the vignettes.

Citation

If you are using scoringutils in your work please consider citing it using the output of citation("scoringutils") (or print(citation("scoringutils"), bibtex = TRUE)):

citation("scoringutils") 

How to make a bug report or feature request

Please briefly describe your problem and what output you expect in an issue. If you have a question, please don't open an issue. Instead, ask on our Q and A page.

Contributing

We welcome contributions and new contributors! We particularly appreciate help on priority problems in the issues. Please check and add to the issues, and/or add a pull request.

Code of Conduct

Please note that the scoringutils project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

epiforecasts/scoringutils documentation built on May 9, 2024, 12:52 a.m.