cov.roc  R Documentation 
This function computes the covariance between the AUC of two correlated (or paired) ROC curves.
cov(...)
## Default S3 method:
cov(...)
## S3 method for class 'auc'
cov(roc1, roc2, ...)
## S3 method for class 'smooth.roc'
cov(roc1, roc2, ...)
## S3 method for class 'roc'
cov(roc1, roc2, method=c("delong", "bootstrap", "obuchowski"),
reuse.auc=TRUE, boot.n=2000, boot.stratified=TRUE, boot.return=FALSE,
progress=getOption("pROCProgress")$name, parallel=FALSE, ...)
roc1, roc2 
the two ROC curves on which to compute the covariance. Either “roc”, “auc” or “smooth.roc” objects (types can be mixed as long as the original ROC curve are paired). 
method 
the method to use, either “delong” or “bootstrap”. The first letter is sufficient. If omitted, the appropriate method is selected as explained in details. 
reuse.auc 
if 
boot.n 
for 
boot.stratified 
for 
boot.return 
if TRUE and 
progress 
the name of progress bar to display. Typically
“none”, “win”, “tk” or “text” (see the

parallel 
if TRUE, the bootstrap is processed in parallel, using parallel backend provided by plyr (foreach). 
... 
further arguments passed to or from other methods,
especially arguments for 
This function computes the covariance between the AUC of two
correlated (or paired, according to the detection of are.paired
) ROC
curves. It is typically called with the two roc objects of
interest. Two methods are available: “delong” and
“bootstrap” (see “Computational
details” section below).
The default is to use “delong” method except with partial AUC and smoothed curves where “bootstrap” is employed. Using “delong” for partial AUC and smoothed ROCs is not supported.
For smoothed ROC curves, smoothing is performed again at each
bootstrap replicate with the parameters originally provided.
If a density smoothing was performed with userprovided
density.cases
or density.controls
the bootstrap cannot
be performed and an error is issued.
cov.default
forces the usage of the
cov
function in the stats package, so
that other code relying on cov
should continue to function
normally.
The numeric value of the covariance.
If boot.return=TRUE
and method="bootstrap"
, an attribute
resampled.values
is set with the resampled (bootstrapped)
values. It contains a matrix with the columns representing the two ROC
curves, and the rows the boot.n
bootstrap replicates.
To compute the covariance of the AUC of the ROC curves, cov
needs a specification of the
AUC. The specification is defined by:
the “auc” field in the “roc” objects if
reuse.auc
is set to TRUE
(default)
passing the specification to auc
with ...
(arguments partial.auc
, partial.auc.correct
and
partial.auc.focus
). In this case, you must ensure either that
the roc
object do not contain an auc
field (if
you called roc
with auc=FALSE
), or set
reuse.auc=FALSE
.
If reuse.auc=FALSE
the auc
function will always
be called with ...
to determine the specification, even if
the “roc” objects do contain an auc
field.
As well if the “roc” objects do not contain an auc
field, the auc
function will always be called with
...
to determine the specification.
Warning: if the roc object passed to roc.test contains an auc
field and reuse.auc=TRUE
, auc is not called and
arguments such as partial.auc
are silently ignored.
With method="bootstrap"
, the processing is done as follow:
boot.n
bootstrap replicates are drawn from the
data. If boot.stratified
is TRUE, each replicate contains
exactly the same number of controls and cases than the original
sample, otherwise if FALSE the numbers can vary.
for each bootstrap replicate, the AUC of the two ROC curves are computed and stored.
the variance (as per var.roc
) of the resampled
AUCs and their covariance are assessed in a single bootstrap pass.
The following formula is used to compute the final covariance:
Var[AUC1] + Var[AUC2]  2cov[AUC1,AUC2]
With method="delong"
, the processing is done as described in
Hanley and HajianTilaki (1997) using the algorithm by Sun and Xu (2014).
With method="obuchowski"
, the processing is done as described
in Obuchowski and McClish (1997), Table 1 and Equation 5, p. 1531. The
computation of g
for partial area under the ROC curve is
modified as:
expr1 * (2 * pi * expr2) ^ {(1)} * (expr4)  A * B * expr1 * (2 * pi * expr2^3) ^ {(1/2)} * expr3
.
The “obuchowski” method makes the assumption that the data is binormal.
If the data shows a deviation from this assumption, it might help to
normalize the data first (that is, before calling roc
),
for example with quantile normalization:
norm.x < qnorm(rank(x)/(length(x)+1)) cov(roc(response, norm.x, ...), ...)
“delong” and “bootstrap” methods make no such assumption.
If density.cases
and density.controls
were provided
for smoothing, the error “Cannot compute the covariance on ROC
curves smoothed with density.controls and density.cases.” is
issued.
If “auc” specifications are different in both roc objects, the warning “Different AUC specifications in the ROC curves. Enforcing the inconsistency, but unexpected results may be produced.” is issued. Unexpected results may be produced.
If one or both ROC curves are “smooth.roc” objects with
different smoothing specifications, the warning
“Different smoothing parameters in the ROC curves. Enforcing
the inconsistency, but unexpected results may be produced.” is issued.
This warning can be benign, especially if ROC curves were generated
with roc(..., smooth=TRUE)
with different arguments to other
functions (such as plot), or if you really want to compare two ROC
curves smoothed differently.
If method="delong"
and the AUC specification specifies a
partial AUC, the warning “Using DeLong for partial AUC is
not supported. Using bootstrap test instead.” is issued. The
method
argument is ignored and “bootstrap” is used instead.
If method="delong"
and the ROC
curve is smoothed, the warning “Using DeLong for
smoothed ROCs is not supported. Using bootstrap instead.” is
issued. The method
argument is ignored and “bootstrap”
is used instead.
DeLong ignores the direction of the ROC curve so that if two
ROC curves have a different direction
, the warning
“"DeLong should not be applied to ROC curves with a different
direction."” is printed. However, the spurious computation is enforced.
If boot.stratified=FALSE
and the sample has a large imbalance between
cases and controls, it could happen that one or more of the replicates
contains no case or control observation, or that there are not enough
points for smoothing, producing a NA
area.
The warning “NA value(s) produced during bootstrap were ignored.”
will be issued and the observation will be ignored. If you have a large
imbalance in your sample, it could be safer to keep
boot.stratified=TRUE
.
When both ROC curves have an auc
of 1 (or 100%), their covariance will always be null.
This is true for both “delong” and “bootstrap” and methods. This result is misleading,
as the covariance is of course not null.
A warning
will be displayed to inform of this condition, and of the misleading output.
The covariance can only be computed on paired data. This
assumption is enforced by are.paired
. If the ROC curves
are not paired, the covariance is 0
and the message “ROC
curves are unpaired.” is printed. If your ROC curves are paired, make
sure they fit are.paired
criteria.
Elisabeth R. DeLong, David M. DeLong and Daniel L. ClarkePearson (1988) “Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach”. Biometrics 44, 837–845.
James A. Hanley and Karim O. HajianTilaki (1997) “Sampling variability of nonparametric estimates of the areas under receiver operating characteristic curves: An update”. Academic Radiology 4, 49–58. DOI: \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1016/S10766332(97)801614")}.
Nancy A. Obuchowski, Donna K. McClish (1997). “Sample size determination for diagnostic accurary studies involving binormal ROC curve indices”. Statistics in Medicine, 16(13), 1529–1542. DOI: \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1002/(SICI)10970258(19970715)16:13<1529::AIDSIM565>3.0.CO;2H")}.
Xu Sun and Weichao Xu (2014) “Fast Implementation of DeLongs Algorithm for Comparing the Areas Under Correlated Receiver Operating Characteristic Curves”. IEEE Signal Processing Letters, 21, 1389–1393. DOI: \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1109/LSP.2014.2337313")}.
Hadley Wickham (2011) “The SplitApplyCombine Strategy for Data Analysis”. Journal of Statistical Software, 40, 1–29. URL: \Sexpr[results=rd]{tools:::Rd_expr_doi("10.18637/jss.v040.i01")}.
roc
, var.roc
CRAN package plyr, employed in this function.
data(aSAH)
# Basic example with 2 roc objects
roc1 < roc(aSAH$outcome, aSAH$s100b)
roc2 < roc(aSAH$outcome, aSAH$wfns)
cov(roc1, roc2)
## Not run:
# The latter used Delong. To use bootstrap:
cov(roc1, roc2, method="bootstrap")
# Decrease boot.n for a faster execution:
cov(roc1, roc2, method="bootstrap", boot.n=1000)
## End(Not run)
# To use Obuchowski:
cov(roc1, roc2, method="obuchowski")
## Not run:
# Comparison can be done on smoothed ROCs
# Smoothing is redone at each iteration, and execution is slow
cov(smooth(roc1), smooth(roc2))
## End(Not run)
# or from an AUC (no smoothing)
cov(auc(roc1), roc2)
## Not run:
# With bootstrap and return.values, one can compute the variances of the
# ROC curves in one single bootstrap run:
cov.rocs < cov(roc1, roc2, method="bootstrap", boot.return=TRUE)
# var(roc1):
var(attr(cov.rocs, "resampled.values")[,1])
# var(roc2):
var(attr(cov.rocs, "resampled.values")[,2])
## End(Not run)
## Not run:
# Covariance of partial AUC:
roc3 < roc(aSAH$outcome, aSAH$s100b, partial.auc=c(1, 0.8), partial.auc.focus="se")
roc4 < roc(aSAH$outcome, aSAH$wfns, partial.auc=c(1, 0.8), partial.auc.focus="se")
cov(roc3, roc4)
# This is strictly equivalent to:
cov(roc3, roc4, method="bootstrap")
# Alternatively, we could reuse roc1 and roc2 to get the same result:
cov(roc1, roc2, reuse.auc=FALSE, partial.auc=c(1, 0.8), partial.auc.focus="se")
## End(Not run)
# Spurious use of DeLong's test with different direction:
roc5 < roc(aSAH$outcome, aSAH$s100b, direction="<")
roc6 < roc(aSAH$outcome, aSAH$s100b, direction=">")
cov(roc5, roc6, method="delong")
## Test data from Hanley and HajianTilaki, 1997
disease.present < c("Yes", "No", "Yes", "No", "No", "Yes", "Yes", "No",
"No", "Yes", "No", "No", "Yes", "No", "No")
field.strength.1 < c(1, 2, 5, 1, 1, 1, 2, 1, 2, 2, 1, 1, 5, 1, 1)
field.strength.2 < c(1, 1, 5, 1, 1, 1, 4, 1, 2, 2, 1, 1, 5, 1, 1)
roc7 < roc(disease.present, field.strength.1)
roc8 < roc(disease.present, field.strength.2)
# Assess the covariance:
cov(roc7, roc8)
## Not run:
# With bootstrap:
cov(roc7, roc8, method="bootstrap")
## End(Not run)
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.