pROC-package: pROC

In pROC: Display and Analyze ROC Curves

pROC

Description

Tools for visualizing, smoothing and comparing receiver operating characteristic (ROC curves). (Partial) area under the curve (AUC) can be compared with statistical tests based on U-statistics or bootstrap. Confidence intervals can be computed for (p)AUC or ROC curves. Sample size / power computation for one or two ROC curves are available.

Details

The basic unit of the pROC package is the roc function. It will build a ROC curve, smooth it if requested (if smooth=TRUE), compute the AUC (if auc=TRUE), the confidence interval (CI) if requested (if ci=TRUE) and plot the curve if requested (if plot=TRUE).

The roc function will call smooth, auc, ci and plot as necessary. See these individual functions for the arguments that can be passed to them through roc. These function can be called separately.

Two paired (that is roc objects with the same response) or unpaired (with different response) ROC curves can be compared with the roc.test function.

Citation

If you use pROC in published research, please cite the following paper:

Xavier Robin, Natacha Turck, Alexandre Hainard, Natalia Tiberti, Frédérique Lisacek, Jean-Charles Sanchez and Markus Müller (2011). “pROC: an open-source package for R and S+ to analyze and compare ROC curves”. BMC Bioinformatics, 12, p. 77. DOI: \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1186/1471-2105-12-77")}

Type citation("pROC") for a BibTeX entry.

The authors would be glad to hear how pROC is employed. You are kindly encouraged to notify Xavier Robin <[e-mail]> about any work you publish.

Abbreviations

The following abbreviations are employed extensively in this package:

• ROC: receiver operating characteristic

• AUC: area under the ROC curve

• pAUC: partial area under the ROC curve

• CI: confidence interval

• SP: specificity

• SE: sensitivity

Functions

roc	Build a ROC curve
are.paired	Dertermine if two ROC curves are paired
auc	Compute the area under the ROC curve
ci	Compute confidence intervals of a ROC curve
ci.auc	Compute the CI of the AUC
ci.coords	Compute the CI of arbitrary coordinates
ci.se	Compute the CI of sensitivities at given specificities
ci.sp	Compute the CI of specificities at given sensitivities
ci.thresholds	Compute the CI of specificity and sensitivity of thresholds
ci.coords	Compute the CI of arbitrary coordinates
coords	Coordinates of the ROC curve
cov	Covariance between two AUCs
ggroc	Plot a ROC curve with ggplot2
has.partial.auc	Determine if the ROC curve have a partial AUC
lines.roc	Add a ROC line to a ROC plot
plot.ci	Plot CIs
plot	Plot a ROC curve
power.roc.test	Sample size and power computation
print	Print a ROC curve object
roc.test	Compare two ROC curves
smooth	Smooth a ROC curve
var	Variance of the AUC

Dataset

This package comes with a dataset of 141 patients with aneurysmal subarachnoid hemorrhage: aSAH.

Installing and using

To install this package, make sure you are connected to the internet and issue the following command in the R prompt:

    install.packages("pROC")
  

To load the package in R:

    library(pROC)
  

Experimental: pipelines

Since version 1.15.0, the roc function can be used in pipelines, for instance with dplyr or magrittr. This is still a highly experimental feature and will change significantly in future versions (see issue 54). The roc.data.frame method supports both standard and non-standard evaluation (NSE), and the roc_ function supports standard evaluation only.

library(dplyr)
aSAH %>% 
    filter(gender == "Female") %>%
    roc(outcome, s100b)
	

By default it returns the roc object, which can then be piped to the coords function to extract coordinates that can be used in further pipelines.

aSAH %>% 
    filter(gender == "Female") %>%
    roc(outcome, s100b) %>%
    coords(transpose=FALSE) %>%
    filter(sensitivity > 0.6, 
           specificity > 0.6)
	

More details and use cases are available in the roc help page.

Bootstrap

All the bootstrap operations for significance testing, confidence interval, variance and covariance computation are performed with non-parametric stratified or non-stratified resampling (according to the stratified argument) and with the percentile method, as described in Carpenter and Bithell (2000) sections 2.1 and 3.3.

Stratification of bootstrap can be controlled with boot.stratified. In stratified bootstrap (the default), each replicate contains the same number of cases and controls than the original sample. Stratification is especially useful if one group has only little observations, or if groups are not balanced.

The number of bootstrap replicates is controlled by boot.n. Higher numbers will give a more precise estimate of the significance tests and confidence intervals but take more time to compute. 2000 is recommanded by Carpenter and Bithell (2000) for confidence intervals. In our experience this is sufficient for a good estimation of the first significant digit only, so we recommend the use of 10000 bootstrap replicates to obtain a good estimate of the second significant digit whenever possible.

Progress bars

Progress bars are no longer supported in pROC 1.19. The use of the progress argument is deprecated, and any value other than NULL will produce a warning. The argument will be removed in a future version.

Handling large datasets

Algorithms

Over the years, a significant amount of time has been invested in making pROC run faster and faster. From the naive algorithm iterating over all thresholds implemented in the first version, we went to a C++ implementation (with Rcpp), and a different algorithm using cummulative sum of responses sorted by the predictor, which scales only with the number of data points, independently on the number of thresholds (algorithm = 2). The curves themselves are identical, but computation time has been decreased massively.

Earlier versions of pROC allowed choosing the optimal algorithm. Since pROC 1.19, only a single algorithm (previously algorithm = 2) is available. This allowed us to remove legacy code and dependencies, and make pROC more maintainable and future proof.

Boostrap

Bootstrap is typically slow because it involves repeatedly computing the ROC curve (or a part of it).

Some bootstrap functions are faster than others. Typically, ci.thresholds is the fastest, and ci.coords the slowest. Use ci.coords only if the CI you need cannot be computed by the specialized CI functions ci.thresholds, ci.se and ci.sp. Note that ci.auc cannot be replaced anyway.

A naive way to speed-up the boostrap is by removing the progress bar:

rocobj <- roc(response, round(predictor))
system.time(ci(rocobj))
system.time(ci(rocobj, progress = "none"))
    

It is of course possible to reduce the number of boostrap iterations. See the boot.n argument to ci. This will reduce the precision of the bootstrap estimate.

Parallel processing

Parallel processing is no longer supported in pROC 1.19. The use of the parallel argument is deprecated, and any value other than FALSE will produce a warning. The argument will be removed in a future version.

Using DeLong instead of boostrap

DeLong is an asymptotically exact method to evaluate the uncertainty of an AUC (DeLong et al. (1988)). Since version 1.9, pROC uses the algorithm proposed by Sun and Xu (2014) which has an O(N log N) complexity and is always faster than bootstrapping. By default, pROC will choose the DeLong method whenever possible.

rocobj <- roc(response, round(predictor), algorithm=3)
system.time(ci(rocobj, method="delong"))
    

Author(s)

Xavier Robin, Natacha Turck, Jean-Charles Sanchez and Markus Müller

Maintainer: Xavier Robin <[e-mail]>

References

James Carpenter and John Bithell (2000) “Bootstrap condence intervals: when, which, what? A practical guide for medical statisticians”. Statistics in Medicine 19, 1141–1164. DOI: \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F")}.

Elisabeth R. DeLong, David M. DeLong and Daniel L. Clarke-Pearson (1988) “Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach”. Biometrics 44, 837–845.

Tom Fawcett (2006) “An introduction to ROC analysis”. Pattern Recognition Letters 27, 861–874. DOI: \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1016/j.patrec.2005.10.010")}.

Xavier Robin, Natacha Turck, Alexandre Hainard, et al. (2011) “pROC: an open-source package for R and S+ to analyze and compare ROC curves”. BMC Bioinformatics, 7, 77. DOI: \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1186/1471-2105-12-77")}.

Xu Sun and Weichao Xu (2014) “Fast Implementation of DeLongs Algorithm for Comparing the Areas Under Correlated Receiver Operating Characteristic Curves”. IEEE Signal Processing Letters, 21, 1389–1393. DOI: \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1109/LSP.2014.2337313")}.

Hadley Wickham (2011) “The Split-Apply-Combine Strategy for Data Analysis”. Journal of Statistical Software, 40, 1–29. URL: \Sexpr[results=rd]{tools:::Rd_expr_doi("10.18637/jss.v040.i01")}.

See Also

CRAN packages ROCR, verification or Bioconductor's roc for ROC curves.

CRAN packages MASS and logcondens employed in this package.

Examples

data(aSAH)

## Build a ROC object and compute the AUC ##
roc1 <- roc(aSAH$outcome, aSAH$s100b)
print(roc1)

# With a formula
roc(outcome ~ s100b, aSAH)
# With pipes, dplyr-style:
## Not run: 
library(dplyr)
aSAH %>% roc(outcome, s100b)
## End(Not run)

# Create a few more curves for the next examples
roc2 <- roc(aSAH$outcome, aSAH$wfns)
roc3 <- roc(aSAH$outcome, aSAH$ndka)


## AUC ##
auc(roc1, partial.auc = c(1, .9))


## Smooth ROC curve ##
smooth(roc1)


## Summary statistics
var(roc1)
cov(roc1, roc3)


## Plot the curve ##
plot(roc1)

#  More plotting options, CI and plotting
# with all-in-one syntax:
roc4 <- roc(aSAH$outcome,
            aSAH$s100b, percent=TRUE,
            # arguments for auc
            partial.auc=c(100, 90), partial.auc.correct=TRUE,
            partial.auc.focus="sens",
            # arguments for ci
            ci=TRUE, boot.n=100, ci.alpha=0.9, stratified=FALSE,
            # arguments for plot
            plot=TRUE, auc.polygon=TRUE, max.auc.polygon=TRUE, grid=TRUE,
            print.auc=TRUE, show.thres=TRUE)

# Add to an existing plot. Beware of 'percent' specification!
roc5 <- roc(aSAH$outcome, aSAH$wfns,
            plot=TRUE, add=TRUE, percent=roc4$percent)


## With ggplot2 ##
if (require(ggplot2)) {
# Create multiple curves to plot
rocs <- roc(outcome ~ wfns + s100b + ndka, data = aSAH)
ggroc(rocs)
}


## Coordinates of the curve ##
coords(roc1, "best", ret=c("threshold", "specificity", "1-npv"))
coords(roc2, "local maximas", ret=c("threshold", "sens", "spec", "ppv", "npv"))


## Confidence intervals ##

# CI of the AUC
ci(roc2)

## Not run: 
# CI of the curve
sens.ci <- ci.se(roc1, specificities=seq(0, 100, 5))
plot(sens.ci, type="shape", col="lightblue")
plot(sens.ci, type="bars")
## End(Not run)

# need to re-add roc2 over the shape
plot(roc2, add=TRUE)

## Not run: 
# CI of thresholds
plot(ci.thresholds(roc2))
## End(Not run)

## Comparisons ##

# Test on the whole AUC
roc.test(roc1, roc2, reuse.auc=FALSE)

## Not run: 
# Test on a portion of the whole AUC
roc.test(roc1, roc2, reuse.auc=FALSE, partial.auc=c(100, 90),
         partial.auc.focus="se", partial.auc.correct=TRUE)

# With modified bootstrap parameters
roc.test(roc1, roc2, reuse.auc=FALSE, partial.auc=c(100, 90),
         partial.auc.correct=TRUE, boot.n=1000, boot.stratified=FALSE)
## End(Not run)


## Power & sample size ##

# Power
# 1 curve
power.roc.test(roc1)
# 2 curves
power.roc.test(roc3, roc2)

# Sample size 
# 1 curve
power.roc.test(roc3, power = 0.9)
# 2 curves
power.roc.test(roc1, roc2, power = 0.9)

# Also without ROC objects.
# For instance what AUC would be significantly different from 0.5?
power.roc.test(ncases=41, ncontrols=72, sig.level=0.05, power=0.95)

pROC documentation built on Aug. 8, 2025, 6:28 p.m.