interval_coverage_deviation: Interval coverage deviation (for quantile-based forecasts)

In epiforecasts/scoringutils: Utilities for Scoring and Assessing Predictions

Interval coverage deviation (for quantile-based forecasts)

Description

Check the agreement between desired and actual interval coverage of a forecast.

The function is similar to interval_coverage(), but takes all provided prediction intervals into account and compares nominal interval coverage (i.e. the desired interval coverage) with the actual observed interval coverage.

A central symmetric prediction interval is defined by a lower and an upper bound formed by a pair of predictive quantiles. For example, a 50% prediction interval is formed by the 0.25 and 0.75 quantiles of the predictive distribution. Ideally, a forecaster should aim to cover about 50% of all observed values with their 50% prediction intervals, 90% of all observed values with their 90% prediction intervals, and so on.

For every prediction interval, the deviation is computed as the difference between the observed interval coverage and the nominal interval coverage For a single observed value and a single prediction interval, coverage is always either 0 or 1 (FALSE or TRUE). This is not the case for a single observed value and multiple prediction intervals, but it still doesn't make that much sense to compare nominal (desired) coverage and actual coverage for a single observation. In that sense coverage deviation only really starts to make sense as a metric when averaged across multiple observations).

Positive values of interval coverage deviation are an indication for underconfidence, i.e. the forecaster could likely have issued a narrower forecast. Negative values are an indication for overconfidence, i.e. the forecasts were too narrow.

\textrm{interval coverage deviation} = \mathbf{1}(\textrm{observed value falls within interval}) - \textrm{nominal interval coverage}

The interval coverage deviation is then averaged across all prediction intervals. The median is ignored when computing coverage deviation.

Usage

interval_coverage_deviation(observed, predicted, quantile_level)

Arguments

observed	Numeric vector of size n with the observed values.
predicted	Numeric nxN matrix of predictive quantiles, n (number of rows) being the number of forecasts (corresponding to the number of observed values) and N (number of columns) the number of quantiles per forecast. If observed is just a single number, then predicted can just be a vector of size N.
quantile_level	Vector of of size N with the quantile levels for which predictions were made.

Value

A numeric vector of length n with the interval coverage deviation for each forecast (with the forecast itself comprising one or multiple prediction intervals).

Examples

observed <- c(1, -15, 22)
predicted <- rbind(
  c(-1, 0, 1, 2, 3),
  c(-2, 1, 2, 2, 4),
  c(-2, 0, 3, 3, 4)
)
quantile_level <- c(0.1, 0.25, 0.5, 0.75, 0.9)
interval_coverage_deviation(observed, predicted, quantile_level)

epiforecasts/scoringutils documentation built on April 23, 2024, 4:56 p.m.