percentile_score | R Documentation |
These accuracy measures can be used to evaluate how accurately a forecast distribution predicts a given actual value.
percentile_score(.dist, .actual, na.rm = TRUE, ...)
quantile_score(
.dist,
.actual,
probs = c(0.05, 0.25, 0.5, 0.75, 0.95),
na.rm = TRUE,
...
)
CRPS(.dist, .actual, n_quantiles = 1000, na.rm = TRUE, ...)
distribution_accuracy_measures
.dist |
The distribution of fitted values from the model, or forecasted values from the forecast. |
.actual |
A vector of responses matching the fitted values
(for forecast accuracy, |
na.rm |
Remove the missing values before calculating the accuracy measure |
... |
Additional arguments for each measure. |
probs |
A vector of probabilities at which the metric is evaluated. |
n_quantiles |
The number of quantiles to use in approximating CRPS when an exact solution is not available. |
An object of class list
of length 2.
A quantile (or percentile) score evaluates how accurately a set of quantiles (or percentiles) from the distribution match the given actual value. This score uses a pinball loss function, and can be calculated via the average of the score function given below:
The score function s_p(q_p,y)
is given by (1-p)(q_p-y)
if
y < q_p
, and p(y-q_p)
if y \ge q_p
. Where p
is the
quantile probability, q_p = F^{-1}(p)
is the quantile with probability
p
, and y
is the actual value.
The resulting accuracy measure will average this score over all predicted
points at all desired quantiles (defined via the probs
argument).
The percentile score is uses the same method with probs
set to all
percentiles probs = seq(0.01, 0.99, 0.01)
.
The continuous ranked probability score (CRPS) is the continuous analogue of the pinball loss quantile score defined above. Its value is twice the integral of the quantile score over all possible quantiles:
CRPS(F,y) = 2 \int_0^1 s_p(q_p,y) dp
It can be computed directly from the distribution via:
CRPS(F,y) = \int_{-\infty}^\infty (F(x) - 1{y\leq x})^2 dx
For some forecast distribution F
and actual value y
.
Calculating the CRPS accuracy measure is computationally difficult for many
distributions, however it can be computed quickly and exactly for Normal and
emperical (sample) distributions. For other distributions the CRPS is
approximated using the quantile score of many quantiles (using the number of
quantiles specified in the n_quantiles
argument).
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.