scores_sample_multiv: Multivariate Scoring Rules for Simulated Forecast...

In scoringRules: Scoring Rules for Parametric and Simulated Distribution Forecasts

Multivariate Scoring Rules for Simulated Forecast Distributions

Description

Compute multivariate scores of the form S(y, dat), where S is a proper scoring rule, y is a d-dimensional realization vector and dat is a simulated sample of multivariate forecasts. Three scores are available: The energy score, a score based on a Gaussian kernel (mmds_sample, see details below) and the variogram score of order p.

Usage

es_sample(y, dat, w = NULL)

mmds_sample(y, dat, w = NULL)

vs_sample(y, dat, w = NULL, w_vs = NULL, p = 0.5)

Arguments

y	realized values (numeric vector of length d).
dat	numeric matrix of data (columns are simulation draws from multivariate forecast distribution).
w	numeric vector of weights for forecast draws (length equal to number of columns of dat)
w_vs	numeric matrix of weights for dat used in the variogram score. This matrix must be square and symmetric, with all elements being non-negative. If no weights are specified, constant weights (with all elements of w_vs equal to one) are used.
p	order of variogram score. Standard choices include p = 1 and p = 0.5.

Details

In the input matrix dat each column is expected to represent a sample from the multivariate forecast distribution, the number of rows of dat thus has to match the length of the observation vector y, and the number of columns of dat is the number of simulated samples.

In es_sample and mmds_sample it is possible to specify a vector w of weights attached to each forecast draw (i.e. each column of matrix dat). These weights are analogous to the input w in crps_sample.

In vs_sample it is possible to specify a matrix w_vs of non-negative weights that allow to emphasize or downweight pairs of component combinations based on subjective expert decisions. w_vs is a square, symmetric matrix with dimensions equal to the length of the input vector y, and the entry in the i-th row and j-th column of w_vs corresponds to the weight assigned to the combination of the corresponding i-th and j-th component. A small example is provided below. For details and further examples, see Scheuerer and Hamill (2015).

The ‘MMD score’ in mmds_sample is a kernel scoring rule as described in Gneiting and Raftery (2007, Section 5). As for all other scores, we use a negative orientation, such that a smaller score corresponds to a better forecast. We use a Gaussian kernel with standard deviation 1. This kernel is the same as the one obtained by setting rbfdot(sigma = .5) in the R package kernlab (Karatzoglou et al., 2004). The naming prefix ‘MMD’ is motivated by the machine learning literature on two sample testing (e.g. Gretton et al., 2012), where this type of kernel function is popular.

Value

Value of the score. A lower score indicates a better forecast.

Author(s)

Maximiliane Graeter, Sebastian Lerch, Fabian Krueger

References

Energy score

Gneiting, T., Stanberry, L.I., Grimit, E.P., Held, L. and N.A. Johnson (2008): 'Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds', TEST, 17, 211-235. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1007/s11749-008-0114-x")}

MMD score

Gneiting, T. and A.E. Raftery (2007): ‘Strictly proper scoring rules, prediction and estimation’, Journal of the American Statistical Association 102, 359-378. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1198/016214506000001437")}

Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B. and A. Smola (2012): ‘A kernel two-sample test’, Journal of' Machine Learning Research, 13, 723-773.

Karatzoglou, A., Smola, A., Hornik, K. and Zeileis A. (2004). kernlab - An S4 Package for Kernel Methods in R. Journal of Statistical Software 11, 1-20. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.18637/jss.v011.i09")}

Variogram-based proper scoring rules

Scheuerer, M. and T.M. Hamill (2015): 'Variogram-based proper scoring rules for probabilistic forecasts of multivariate quantities', Monthly Weather Review, 143, 1321-1334. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1175/mwr-d-14-00269.1")}

See Also

scores_sample_multiv_weighted for weighted versions of the scoring rules documented here.

Examples

d <- 10  # number of dimensions
m <- 50  # number of samples from multivariate forecast distribution

# parameters for multivariate normal example
mu0 <- rep(0, d)
mu <- rep(1, d)
S0 <- S <- diag(d)
S0[S0==0] <- 0.2
S[S==0] <- 0.1

# generate samples from multivariate normal distributions
obs <- drop(mu0 + rnorm(d) %*% chol(S0))
fc_sample <- replicate(m, drop(mu + rnorm(d) %*% chol(S)))

# compute Energy Score
es_sample(y = obs, dat = fc_sample)

# in the univariate case, Energy Score and CRPS are the same
# illustration: Evaluate forecast sample for the first variable
es_sample(y = obs[1], dat = fc_sample[1, , drop = FALSE])
crps_sample(y = obs[1], dat = fc_sample[1, ])

# illustration of observation weights for Energy Score
# example: equal weights for first half of draws; zero weights for other draws
w <- rep(c(1, 0), each = .5*m)/(.5*m)
es_sample(y = obs, dat = fc_sample, w = w)

# weighting matrix for variogram score
# note that, unlike for w, weights in w_vs refer to dimensions 
# (rows of dat) rather than draws (cols of dat)
w_vs <- outer(1:d, 1:d, function(x, y) .5^abs(x-y))

vs_sample(y = obs, dat = fc_sample) 
vs_sample(y = obs, dat = fc_sample, w_vs = w_vs) 
vs_sample(y = obs, dat = fc_sample, w_vs = w_vs, p = 1)

scoringRules documentation built on May 31, 2023, 6:06 p.m.