brierscore: Calculate Brier Scores And Decompositions

In scoring: Proper Scoring Rules

Description

Calculate Brier scores, average Brier scores by a grouping variable, and Brier score decompositions for two-alternative forecasts.

Usage

brierscore(object, data, group = NULL, decomp = FALSE, bounds = NULL,
           reverse = FALSE, wt = NULL, decompControl = list())

Arguments

object	an object of class "formula", of the form outcome ~ forecast. See calcscore() documentation for further details.
data	an optional data frame or list containing the variables in the formula. If not found in data, the variables are taken from the environment from which calcscore is called.
group	the name of a grouping variable within data, which is used to calculate average Brier score by group.
decomp	if TRUE, Brier score decompositions are calculated.
bounds	a vector of length 2 corresponding to the desired minimum and maximum Brier score, respectively.
reverse	if FALSE (default), smaller scores imply better forecasts. If TRUE, larger scores imply better forecasts.
wt	a vector of weights for computing a weighted Brier score. If NULL, the weights are set to 1/n, where n is the number of forecasts (this corresponds to a simple average Brier score).
decompControl	a list of additional settings for the Brier decomposition. See options below.

Details

If decomp=TRUE or group is supplied, the function returns a list (see value section). Otherwise, the function returns a numeric vector containing the Brier score associated with each forecast. Abbreviations in the output include discrim (discrimination; also called resolution), miscal (miscalibration; also called reliability or calibration), miscal_lg (miscalibration in the large), and unc (outcome uncertainty). Formal definitions of these quantities can be found in Table 1 of the Merkle & Hartman paper referenced below (also see Section 2 of that paper).

Some decompControl arguments are specifically designed for forecasting tournaments and may not be useful in other situations. Possible arguments for decompControl include:

wt

A vector of weights, for performing a weighted Brier decomposition (could also use the simple wt argument).

qid

A vector of question ids, for use with the qtype argument.

bin

If TRUE (default), forecasts are binned prior to decomposition. If FALSE, the original forecasts are maintained.

qtype

A data frame with columns qid, ord, squo. For each unique question id in the qid argument above, this describes whether or not the question is ordinal (1=yes,0=no) and whether or not the question has a "status quo" interpretation (1=yes,0=no).

scale

Should Brier components be rescaled, such that 1 is always best and 0 is always worst? Defaults to FALSE.

roundto

To what value should forecasts be rounded (necessary for Murphy decomposition)? Defaults to .1, meaning that forecasts are rounded to the nearest .1.

binstyle

Method for ensuring that each forecast sums to 1. If equal to 1 (default), the smallest forecast is one minus the sum of the other forecasts. If equal to 2, the forecast furthest from its rounded value is one minus the sum of other forecasts.

resamples

Desired number of Brier resamples (useful for questions with inconsistent alternatives). Defaults to 0; see Merkle & Hartman reference for more detail.

Value

Depending on input arguments, brierscore may return an object of class numeric containing raw Brier scores. It may also return a list containing the objects below.

rawscores	an object of class numeric containing raw Brier scores for each forecast.
brieravg	an object of class numeric containing average Brier scores for each unique value of group. If wt was supplied, this is a weighted sum. Otherwise, it is a simple average (equal weights summing to 1).
decomp	an object of class matrix containing Brier score decompositions and mean Brier scores for each unique value of group.

Author(s)

Ed Merkle

References

Merkle, E. C. & Hartman, R. (2018). Weighted Brier score decompositions for topically heterogenous forecasting tournaments. Judgment and Decision Making, 13, 185-201.

Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78, 1-3.

Murphy, A. H. (1973). A new vector partition of the probability score. Journal of Applied Meteorology, 12, 595-600.

Yates, J. F. (1982). External correspondence: Decompositions of the mean probability score. Organizational Behavior and Human Performance, 30, 132-156.

Young, R. M. B. (2010). Decomposition of the Brier score for weighted forecast-verification pairs. Quarterly Journal of the Royal Meteorological Society, 136, 1364-1370.

See Also

calcscore

Examples

data("WorldEvents")
## Raw Brier scores
brier1 <- brierscore(answer ~ forecast, data=WorldEvents)
## Raw Brier scores plus group means and decompositions
brier2 <- brierscore(answer ~ forecast, data=WorldEvents,
                     group="forecaster", decomp=TRUE)
## Obtain Brier scores via calcscore
brier3 <- calcscore(answer ~ forecast, data=WorldEvents,
                    param=2, fam="pow")
all.equal(brier1, brier3)

Example output

[1] TRUE

scoring documentation built on May 2, 2019, 4:53 p.m.