quantile_score: Quantile Score

In scoringutils: Utilities for Scoring and Assessing Predictions

Quantile Score

Description

Proper Scoring Rule to score quantile predictions. Smaller values are better. The quantile score is closely related to the Interval score (see interval_score()) and is the quantile equivalent that works with single quantiles instead of central prediction intervals.

Usage

quantile_score(true_values, predictions, quantiles, weigh = TRUE)

Arguments

true_values	A vector with the true observed values of size n
predictions	nxN matrix of predictive samples, n (number of rows) being the number of data points and N (number of columns) the number of Monte Carlo samples. Alternatively, predictions can just be a vector of size n.
quantiles	vector of size n with the quantile values of the corresponding predictions.
weigh	if TRUE, weigh the score by alpha / 2, so it can be averaged into an interval score that, in the limit, corresponds to CRPS. Alpha is the value that corresponds to the (alpha/2) or (1 - alpha/2) quantiles provided and will be computed from the quantile. Alpha is the decimal value that represents how much is outside a central prediction interval (E.g. for a 90 percent central prediction interval, alpha is 0.1). Default: TRUE.

Value

vector with the scoring values

References

Strictly Proper Scoring Rules, Prediction,and Estimation, Tilmann Gneiting and Adrian E. Raftery, 2007, Journal of the American Statistical Association, Volume 102, 2007 - Issue 477

Evaluating epidemic forecasts in an interval format, Johannes Bracher, Evan L. Ray, Tilmann Gneiting and Nicholas G. Reich, https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1008618

Examples

true_values <- rnorm(10, mean = 1:10)
alpha <- 0.5

lower <- qnorm(alpha / 2, rnorm(10, mean = 1:10))
upper <- qnorm((1 - alpha / 2), rnorm(10, mean = 1:10))

qs_lower <- quantile_score(true_values,
  predictions = lower,
  quantiles = alpha / 2
)
qs_upper <- quantile_score(true_values,
  predictions = upper,
  quantiles = 1 - alpha / 2
)
interval_score <- (qs_lower + qs_upper) / 2

scoringutils documentation built on Feb. 16, 2023, 7:30 p.m.