score | R Documentation |
This function allows automatic scoring of forecasts using a range of metrics. For most users it will be the workhorse for scoring forecasts as it wraps the lower level functions package functions. However, these functions are also available if you wish to make use of them independently.
A range of forecasts formats are supported, including quantile-based,
sample-based, binary forecasts. Prior to scoring, users may wish to make use
of check_forecasts()
to ensure that the input data is in a supported
format though this will also be run internally by score()
. Examples for
each format are also provided (see the documentation for data
below or in
check_forecasts()
).
Each format has a set of required columns (see below). Additional columns may be present to indicate a grouping of forecasts. For example, we could have forecasts made by different models in various locations at different time points, each for several weeks into the future. It is important, that there are only columns present which are relevant in order to group forecasts. A combination of different columns should uniquely define the unit of a single forecast, meaning that a single forecast is defined by the values in the other columns. Adding additional unrelated columns may alter results.
To obtain a quick overview of the currently supported evaluation metrics,
have a look at the metrics data included in the package. The column
metrics$Name
gives an overview of all available metric names that can be
computed. If interested in an unsupported metric please open a feature request or consider
contributing a pull request.
For additional help and examples, check out the Getting Started Vignette.
score(data, metrics = NULL, ...)
data |
A data.frame or data.table with the predictions and observations.
For scoring using
For scoring integer and continuous forecasts a
For scoring predictions in a quantile-format forecast you should provide
a column called
In addition a You can check the format of your data using |
metrics |
the metrics you want to have in the output. If |
... |
additional parameters passed down to |
A data.table with unsummarised scores. There will be one score per
quantile or sample, which is usually not desired, so you should almost
always run summarise_scores()
on the unsummarised scores.
Nikos Bosse nikosbosse@gmail.com
Funk S, Camacho A, Kucharski AJ, Lowe R, Eggo RM, Edmunds WJ (2019) Assessing the performance of real-time epidemic forecasts: A case study of Ebola in the Western Area region of Sierra Leone, 2014-15. PLoS Comput Biol 15(2): e1006785. doi: 10.1371/journal.pcbi.1006785
library(magrittr) # pipe operator data.table::setDTthreads(1) # only needed to avoid issues on CRAN check_forecasts(example_quantile) score(example_quantile) %>% add_coverage(by = c("model", "target_type")) %>% summarise_scores(by = c("model", "target_type")) # forecast formats with different metrics ## Not run: score(example_binary) score(example_quantile) score(example_integer) score(example_continuous) ## End(Not run) # score point forecasts (marked by 'NA' in the quantile column) score(example_point) %>% summarise_scores(by = "model", na.rm = TRUE)
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.