verify: Verification function

In verification: Weather Forecast Verification Utilities

Description

Based on the type of inputs, this function calculates a range of verification statistics and skill scores. Additionally, it creates a verify class object that can be used in further analysis or with other methods such as plot and summary.

Usage

verify(obs, pred, p = NULL, baseline = NULL, 
    frcst.type = "prob", obs.type = "binary",
    thresholds = seq(0,1,0.1), show = TRUE, bins = TRUE,
    fudge = 0.01, ...)

       

Arguments

obs	The values with which the verifications are verified. May be a vector of length 4 if the forecast and predictions are binary data summarized in a contingency table. In this case, the value are entered in the order of c(n11, n01, n10, n00). If obs is a matrix, it is assumed to be a contingency table with observed values summarized in the columns and forecasted values summarized in the rows.
pred	Prediction of event. The prediction may be in the form of the a point prediction or the probability of a forecast. Let pred = NULL if obs is a contingency table.
p	the probability level of the quantile forecast, any value between 0 and 1.
baseline	In meteorology, climatology is the baseline that represents the no-skill forecast. In other fields this field would differ. This field is used to calculate certain skill scores. If left NULL, these statistics are calculated using sample climatology. If this is not NULL, the mean of these values is used as the baseline forecast. This interpretation is not appropriate for all applications. For example, if a baseline forecast is different for each forecast this will not work appropriately.

frcst.type	Forecast type. One of "prob", "binary", "norm.dist", "cat" or "cont", or "quantile". Defaults to "prob". "norm.dist" is used when the forecast is in the form of a normal distribution. See crps for more details.
obs.type	Observation type. Either "binary", "cat" or "cont". Defaults to "binary"
thresholds	Thresholds to be considered for point forecasts of continuous events.
show	Binary; if TRUE (the default), print warning message
bins	Binary; if TRUE (default), the probabilistic forecasts are placed in bins defined by the sequence defined in threshold and assigned the midpoint value.
fudge	A numeric fudge factor to be added to each cell of the contingency table in order to avoid division by zero.
...	Additional options.

Details

See Wilks (2006) and the WMO Joint WWRP/WGNE Working Group web site on verification for more details about these verification statistics. See Stephenson et al. (2008) and Ferro and Stephenson (2011) for more on the extreme dependence scores and indices. For information on confidence intervals for these scores, see Gilleland (2010).

Value

An object of the verify class. Depending on the type of data used, the following information may be returned. The following notation is used to describe which values are produced for which type of forecast/observations. (BB = binary/binary, PB = probablistic/binary, CC = continuous/continuous, CTCT = categorical/categorical)

BS	Brier Score (PB)
BSS	Brier Skill Score(PB)
SS	Skill Score (BB)
hit.rate	Hit rate, aka PODy, $h$ (PB, CTCT)
false.alarm.rate	False alarm rate, PODn, $f$ (PB, CTCT)
TS	Threat Score or Critical Success Index (CSI)(BB, CTCT)
ETS	Equitable Threat Score (BB, CTCT)
BIAS	Bias (BB, CTCT)
PC	Percent correct or hit rate (BB, CTCT)
Cont.Table	Contingency Table (BB)
HSS	Heidke Skill Score(BB, CTCT)
KSS	Kuniper Skill Score (BB)
PSS	Pierce Skill Score (CTCT)
GS	Gerrity Score (CTCT)
ME	Mean error (CC)
MSE	Mean-squared error (CC)
MAE	Mean absolute error (CC)
theta	Odds Ratio (BB)
log.theta	Log Odds Ratio
n.h	Degrees of freedom for log.theta (BB)
orss	Odds ratio skill score, aka Yules's Q (BB)
eds	Extreme Dependency Score (BB)
eds.se	Standard Error for Extreme Dependence Score (BB)
seds	Symmetric Extreme Dependency Score (BB)
seds.se	Standard Error for Symmetric Extreme Dependency Score (BB)
EDI	Extremal Dependence Index (BB)
EDI.se	Standard Error for Extremal Dependence Index (BB)
SEDI	Symmetric Extremal Dependence Index (BB)
SEDI.se	Standard Error for Symmetric Extremal Dependence Index (BB)

Note

There are other packages in R and Bioconductor which are usefull for verification tasks. This includes the ROCR, ROC, package and the limma package (in the Bioconductor repository.) Written by people in different fields, each provides tools for verification from different perspectives.

For the categorical forecast and verification, the Gerrity score only makes sense for forecast that have order, or are basically ordinal. It is assumed that the forecasts are listed in order. For example, if the rows of a contigency table were summarized as "medium, low, high", the Gerrity score will be incorrectly summarized.

As of version 1.37, the intensity scale (IS) verification funcitons have been removed from this package. Please use SpatialVx for this functionality.

Author(s)

Matt Pocernich

References

Ferro, C. A. T. and D. B. Stephenson, 2011. Extremal dependence indices: Improved verification measures for deterministic forecasts of rare binary events. Wea. Forecasting, 26, 699 - 713.

Gilleland, E., 2010. Confidence intervals for forecast verification. NCAR Technical Note NCAR/TN-479+STR, 71pp. Available at: http://nldr.library.ucar.edu/collections/technotes/asset-000-000-000-846.pdf

Stephenson, D. B., B. Casati, C. A. T. Ferro, and C. A. Wilson, 2008. The extreme dependency score: A non-vanishing measure for forecasts of rare events. Meteor. Appl., 15, 41 - 50.

Wilks, D. S., 2006. Statistical Methods in the Atmospheric Sciences , San Diego: Academic Press., 627 pp. (2nd Editiion).

WMO Joint WWRP/WGNE Working Group on Verification Website

http://www.cawcr.gov.au/projects/verification/

See Also

table.stats

Examples

# binary/binary example
obs<- round(runif(100))
pred<- round(runif(100))

# binary/binary example
# Finley tornado data.

obs<- c(28, 72, 23, 2680)
A<- verify(obs, pred = NULL, frcst.type = "binary", obs.type = "binary")

summary(A)

# categorical/categorical example
# creates a simulated 5 category forecast and observation.
obs <- round(runif(100, 1,5) )
pred <- round(runif(100, 1,5) )

A<- verify(obs, pred, frcst.type = "cat", obs.type = "cat" )
summary(A)

#  probabilistic/ binary example

pred<- runif(100)
A<- verify(obs, pred, frcst.type = "prob", obs.type = "binary")
summary(A)

# continuous/ continuous example
obs<- rnorm(100)
pred<- rnorm(100)
baseline <- rnorm(100, sd = 0.5) 

A<- verify(obs, pred, baseline = baseline,  frcst.type = "cont", obs.type = "cont")
summary(A)

Example output

Loading required package: fields
Loading required package: spam
Loading required package: dotCall64
Loading required package: grid
Spam version 2.2-2 (2019-03-07) is loaded.
Type 'help( Spam)' or 'demo( spam)' for a short introduction 
and overview of this package.
Help for individual functions is also obtained by adding the
suffix '.spam' to the function name, e.g. 'help( chol.spam)'.

Attaching package: 'spam'

The following objects are masked from 'package:base':

    backsolve, forwardsolve

Loading required package: maps
See https://github.com/NCAR/Fields for
 an extensive vignette, other supplements and source code 
Loading required package: boot
Loading required package: CircStats
Loading required package: MASS
Loading required package: dtw
Loading required package: proxy

Attaching package: 'proxy'

The following object is masked from 'package:spam':

    as.matrix

The following objects are masked from 'package:stats':

    as.dist, dist

The following object is masked from 'package:base':

    as.matrix

Loaded dtw v1.20-1. See ?dtw for help, citation("dtw") for use in publication.

[1] " Assume data entered as c(n11, n01, n10, n00) Obs*Forecast"

The forecasts are binary, the observations are binary.
The contingency table for the forecast 
     [,1] [,2]
[1,]   28   72
[2,]   23 2680

PODy =  0.5489 
Std. Err. for POD =  0.06967 
TS   =  0.2276 
Std. Err. for TS =  0.03278 
ETS  =  0.216 
Std. Err. for ETS =  0.03411 
FAR  =  0.7199 
Std. Err. for FAR =  0.03469 
HSS  =  0.3553 
Std. Err. for HSS =  0.04614 
PC   =  0.9661 
Std. Err. for PC =  0.003245 
BIAS =  1.961 
Odds Ratio =  45.31 
Log Odds Ratio =  3.814 
Std. Err. for log Odds Ratio =  0.3057 
Odds Ratio Skill Score =  0.9568 
Std. Err. for Odds Ratio Skill Score =   
Extreme Dependency Score (EDS) =  0.7396 
Std. Err. for EDS =  0.04794 
Symmetric Extreme Dependency Score (SEDS) =  0.5935 
Std. Err. for SEDS =  0.04391 
Extremal Dependence Index (EDI) =  0.7173 
Std. Err. for EDI =  0.06167 
Symmetric Extremal Dependence Index (SEDI) =  0.7527 
Std. Err. for SEDI =  0.06043 

The forecasts are categorical, the observations are categorical.
Percent Correct      =  0.18 
Heidke Skill Score   =  -0.0708 
Pierce Skill Score   =  -0.0706 
Gerrity Score        =  -0.0652 

 Statistics considering each category in turn. 
 
Threat Score                 0 0.0769 0.163 0.109 0.04 
Bias by cat.               
Percent correct by cat.   0.86 0.64 0.59 0.51 0.76 
Hit Rate (POD) by cat.       0 0.143 0.25 0.25 0.0714 
False Alarm Rate by cat.  0.0549 0.228 0.25 0.408 0.128 
False Alarm Ratio by cat.    1 0.857 0.68 0.838 0.917 
If baseline is not included, baseline values  will be calculated from the  sample obs. 

The forecasts are probabilistic, the observations are binary.
Sample baseline calculated from observations.
Brier Score (BS)           =  8.664 
Brier Score - Baseline     =  1.353 
Skill Score                =  -5.403 
Reliability                =  7.403 
Resolution                 =  0.09245 
Uncertainty              =  -6.667 

The forecasts are continuous, the observations are continous.
Baseline data provided. 
MAE               =   1.04 
ME                =  -0.1557 
MSE               =   1.81 
MSE - baseline    =  1.087 
MSE - persistence =  2.385 
SS  - baseline     =  -0.6655 

verification documentation built on May 2, 2019, 1:24 a.m.