scores_sample_multiv | R Documentation |
Compute multivariate scores of the form S(y, dat)
, where S
is a
proper scoring rule, y
is a d-dimensional realization vector and
dat
is a simulated sample of multivariate forecasts. Three scores
are available: The energy score, a score based on a Gaussian kernel
(mmds_sample, see details below) and the variogram score of order p
.
es_sample(y, dat, w = NULL)
mmds_sample(y, dat, w = NULL)
vs_sample(y, dat, w = NULL, w_vs = NULL, p = 0.5)
y |
realized values (numeric vector of length d). |
dat |
numeric matrix of data (columns are simulation draws from multivariate forecast distribution). |
w |
numeric vector of weights for forecast draws (length equal to number of columns of |
w_vs |
numeric matrix of weights for |
p |
order of variogram score. Standard choices include |
In the input matrix dat
each column is expected to represent a sample
from the multivariate forecast distribution, the number of rows of dat
thus has to match the length of the observation vector y
, and the
number of columns of dat
is the number of simulated samples.
In es_sample and mmds_sample it is possible to specify a vector w
of weights
attached to each forecast draw (i.e. each column of matrix dat
). These
weights are analogous to the input w
in crps_sample.
In vs_sample it is possible to specify a matrix w_vs
of
non-negative weights that allow to emphasize or downweight pairs of
component combinations based on subjective expert decisions. w_vs
is a
square, symmetric matrix with dimensions equal to the length of the input vector
y
, and the entry in the i
-th row and j
-th column of
w_vs
corresponds to the weight assigned to the combination of the
corresponding i
-th and j
-th component. A small example is provided below.
For details and further examples, see Scheuerer and Hamill (2015).
The ‘MMD score’ in mmds_sample is a kernel scoring rule as described in
Gneiting and Raftery (2007, Section 5). As for all other scores,
we use a negative orientation, such that a smaller score corresponds to a better
forecast. We use a Gaussian kernel with standard deviation 1. This kernel is
the same as the one obtained by setting rbfdot(sigma = .5)
in the
R package kernlab (Karatzoglou et al., 2004). The naming prefix ‘MMD’ is
motivated by the machine learning literature on two sample testing
(e.g. Gretton et al., 2012), where this type of kernel function is popular.
Value of the score. A lower score indicates a better forecast.
Maximiliane Graeter, Sebastian Lerch, Fabian Krueger
Energy score
Gneiting, T., Stanberry, L.I., Grimit, E.P., Held, L. and N.A. Johnson (2008): 'Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds', TEST, 17, 211-235. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1007/s11749-008-0114-x")}
MMD score
Gneiting, T. and A.E. Raftery (2007): ‘Strictly proper scoring rules, prediction and estimation’, Journal of the American Statistical Association 102, 359-378. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1198/016214506000001437")}
Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B. and A. Smola (2012): ‘A kernel two-sample test’, Journal of' Machine Learning Research, 13, 723-773.
Karatzoglou, A., Smola, A., Hornik, K. and A. Zeileis (2004). kernlab - An S4 Package for Kernel Methods in R. Journal of Statistical Software 11, 1-20. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.18637/jss.v011.i09")}
Variogram-based proper scoring rules
Scheuerer, M. and T.M. Hamill (2015): 'Variogram-based proper scoring rules for probabilistic forecasts of multivariate quantities', Monthly Weather Review, 143, 1321-1334. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1175/mwr-d-14-00269.1")}
scores_sample_multiv_weighted
for weighted versions of the scoring rules documented here.
d <- 10 # number of dimensions
m <- 50 # number of samples from multivariate forecast distribution
# parameters for multivariate normal example
mu0 <- rep(0, d)
mu <- rep(1, d)
S0 <- S <- diag(d)
S0[S0==0] <- 0.2
S[S==0] <- 0.1
# generate samples from multivariate normal distributions
obs <- drop(mu0 + rnorm(d) %*% chol(S0))
fc_sample <- replicate(m, drop(mu + rnorm(d) %*% chol(S)))
# compute Energy Score
es_sample(y = obs, dat = fc_sample)
# in the univariate case, Energy Score and CRPS are the same
# illustration: Evaluate forecast sample for the first variable
es_sample(y = obs[1], dat = fc_sample[1, , drop = FALSE])
crps_sample(y = obs[1], dat = fc_sample[1, ])
# illustration of observation weights for Energy Score
# example: equal weights for first half of draws; zero weights for other draws
w <- rep(c(1, 0), each = .5*m)/(.5*m)
es_sample(y = obs, dat = fc_sample, w = w)
# weighting matrix for variogram score
# note that, unlike for w, weights in w_vs refer to dimensions
# (rows of dat) rather than draws (cols of dat)
w_vs <- outer(1:d, 1:d, function(x, y) .5^abs(x-y))
vs_sample(y = obs, dat = fc_sample)
vs_sample(y = obs, dat = fc_sample, w_vs = w_vs)
vs_sample(y = obs, dat = fc_sample, w_vs = w_vs, p = 1)
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.