roc.plot: Relative operating characteristic curve.

In verification: Weather Forecast Verification Utilities

Description

This function creates Receiver Operating Characteristic (ROC) plots for one or more models. A ROC curve plots the false alarm rate against the hit rate for a probablistic forecast for a range of thresholds. The area under the curve is viewed as a measure of a forecast's accuracy. A measure of 1 would indicate a perfect model. A measure of 0.5 would indicate a random forecast.

Usage

    ## Default S3 method:
roc.plot(x, pred, thresholds = NULL, binormal =
FALSE,   legend = FALSE, leg.text = NULL,  plot = "emp", CI = FALSE,
n.boot = 1000, alpha = 0.05, tck = 0.01, plot.thres = seq(0.1,
0.9, 0.1), show.thres = TRUE, main = "ROC Curve", xlab = "False Alarm Rate",
ylab = "Hit Rate", extra = FALSE,  ...)
## S3 method for class 'prob.bin'
roc.plot(x, ...) 

Arguments

x	A binary observation (coded {0, 1 } ) or a verification object.
pred	A probability prediction on the interval [0,1]. If multiple models are compared, this may be a matrix where each column represents a different prediction.
thresholds	Thresholds may be provided. These thresholds will be used to calculate the hit rate ($h$) and false alarm rate ($f$). If thresholds is NULL, all unique thresholds are used as a threshold. Alternatively, if the number of bins is specified, thresholds will be calculated using the specified numbers of quantiles.

binormal	If TRUE, in addition to the empirical ROC curve, the binormal ROC curve will be calculated. To get a plot draw, plot must be either “binorm” or “both”.
legend	Binomial. Defaults to FALSE indicating whether a legend should be displayed.
leg.text	Character vector for legend. If NULL, models are labeled “Model A", “Model B",...
plot	Either “emp” (default), “binorm” or “both” to determine which plot is shown. If set to NULL, a plot is not created
CI	Confidence Intervals. Calculated by bootstrapping the observations and prediction, then calculating PODy and PODn values.
n.boot	Number of bootstrap samples.
alpha	Confidence interval. By default = 0.05
tck	Tick width on confidence interval whiskers.
plot.thres	By default, displays the threshold levels on the ROC diagrams. To surpress these values, set it equal to NULL. If confidence intervals (CI) is set to TRUE, levels specified here will determine where confidence interval boxes are placed.
show.thres	Show thresholds for points indicated by plot.thres. Defaults to TRUE.
main	Title for plot.
xlab, ylab	Plot axes labels. Defaults to “Hit Rate” and “False Alarm Rate”, for the y and x axes respectively.
extra	Extra text describing binormal and empirical lines.
...	Additional plotting options.

Value

If assigned to an object, the following values are reported.

plot.data	The data used to generate the ROC plots. This is a array. Column headers are thresholds, empirical hit and false alarm rates, and binormal hit and false alarm rates. Each model is depicted on an array indexed by the third dimension.
roc.vol	The areas under the ROC curves. By default,this is printed on the plots. Areas and p-values are calculated with and without adjustments for ties along with the p-value for the area. These values are calculated using roc.area. The fifth column contains the area under the binormal curve, if binormal is selected.
A.boot	If confidence intervals are calculated, the area under the ROC curve are returned.

Note

Other packages in R provide functions to create ROC diagrams and different diagnostics. The ROCR package provides excellent functions to generate ROC diagrams with lines coded by threshold. Large datasets are handled by a sampling routine and the user may plot a number of threshold dependent, contingency table scores. Arguably, this is a superior package with respect to ROC plotting.

There is not a minimum size required to create confidence limits or show thresholds. When there are few data points, it is possilbe to make some pretty unattractive graphs.

The roc.plot method can be used to summarize a "verify, prob.bin" class object created with the verify command. It is appropriate to use the roc plot for forecast which are not probabilities, but rather forecasts made on a continuous scale. The roc plot function can be used to summarize such forecasts but it is not possible to use the verify function to summarize such forecasts. An example is shown below.

Author(s)

Matt Pocernich

References

Mason, I. (1982) “A model for assessment of weather forecasts,” Aust. Met. Mag 30 (1982) 291-303.

Mason, S.J. and N.E. Graham. (2002) “Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation, ” Q. J. R. Meteorol. Soc. 128 pp. 2145-2166.

Swets, John A. (1996) Signal Detection Theory and ROC Analysis in Psychology and Diagnostics, Lawrence Erlbaum Associates, Inc.

See Also

pop and lines.roc

Examples

# Data from Mason and Graham article.

a<- c(0,0,0,1,1,1,0,1,1,0,0,0,0,1,1)
b<- c(.8, .8, 0, 1,1,.6, .4, .8, 0, 0, .2, 0, 0, 1,1)
c<- c(.928,.576, .008, .944, .832, .816, .136, .584, .032, .016, .28, .024, 0, .984, .952)

A<- data.frame(a,b,c)
names(A)<- c("event", "p1", "p2")

## for model with ties
roc.plot(A$event, A$p1)

## for model without ties
roc.plot(A$event, A$p2)

### show binormal curve fit.

roc.plot(A$event, A$p2, binormal = TRUE)
## Not run: 
# icing forecast

data(prob.frcs.dat)
A <- verify(prob.frcs.dat$obs, prob.frcs.dat$frcst/100)
roc.plot(A, main = "AWG Forecast")


# plotting a ``prob.bin'' class object.
obs<- round(runif(100))
pred<- runif(100)

A<- verify(obs, pred, frcst.type = "prob", obs.type = "binary")

roc.plot(A, main = "Test 1", binormal = TRUE, plot = "both")

## show confidence intervals.  MAY BE SLOW
roc.plot(A, threshold = seq(0.1,0.9, 0.1), main = "Test 1", CI = TRUE,
alpha = 0.1)

###   example from forecast verification website. 
data(pop)
d <- pop.convert() ## internal function used to make binary observations for the pop figure.
### note the use of bins = FALSE !!
 mod24 <- verify(d$obs_norain, d$p24_norain, bins = FALSE)

 mod48 <- verify(d$obs_norain, d$p48_norain, bins = FALSE)

roc.plot(mod24, plot.thres = NULL)
lines.roc(mod48, col = 2, lwd = 2)
leg.txt <- c("24 hour forecast", "48 hour forecast")
legend( 0.6, 0.4, leg.txt, col = c(1,2), lwd = 2)

## End(Not run)

Example output

Loading required package: fields
Loading required package: spam
Loading required package: dotCall64
Loading required package: grid
Spam version 2.2-2 (2019-03-07) is loaded.
Type 'help( Spam)' or 'demo( spam)' for a short introduction 
and overview of this package.
Help for individual functions is also obtained by adding the
suffix '.spam' to the function name, e.g. 'help( chol.spam)'.

Attaching package: 'spam'

The following objects are masked from 'package:base':

    backsolve, forwardsolve

Loading required package: maps
See https://github.com/NCAR/Fields for
 an extensive vignette, other supplements and source code 
Loading required package: boot
Loading required package: CircStats
Loading required package: MASS
Loading required package: dtw
Loading required package: proxy

Attaching package: 'proxy'

The following object is masked from 'package:spam':

    as.matrix

The following objects are masked from 'package:stats':

    as.dist, dist

The following object is masked from 'package:base':

    as.matrix

Loaded dtw v1.20-1. See ?dtw for help, citation("dtw") for use in publication.

Warning message:
In wilcox.test.default(pred[obs == 1], pred[obs == 0], alternative = "great") :
  cannot compute exact p-value with ties
If baseline is not included, baseline values  will be calculated from the  sample obs. 
If baseline is not included, baseline values  will be calculated from the  sample obs. 
If baseline is not included, baseline values  will be calculated from the  sample obs. 
If baseline is not included, baseline values  will be calculated from the  sample obs. 

verification documentation built on May 2, 2019, 1:24 a.m.