# ci.thresholds: Compute the confidence interval of thresholds In pROC: Display and Analyze ROC Curves

 ci.thresholds R Documentation

## Compute the confidence interval of thresholds

### Description

This function computes the confidence interval (CI) of the sensitivity and specificity of the thresholds given in argument. By default, the 95% CI are computed with 2000 stratified bootstrap replicates.

### Usage

``````# ci.thresholds(...)
## S3 method for class 'roc'
ci.thresholds(roc, conf.level=0.95, boot.n=2000,
boot.stratified=TRUE, thresholds = "local maximas",
progress=getOption("pROCProgress")\$name, parallel=FALSE, ...)
## S3 method for class 'formula'
ci.thresholds(formula, data, ...)
## S3 method for class 'smooth.roc'
ci.thresholds(smooth.roc, ...)
## Default S3 method:
ci.thresholds(response, predictor, ...)
``````

### Arguments

 `roc` a “roc” object from the `roc` function. `smooth.roc` not available for smoothed ROC curves, available only to catch the error and provide a clear error message. `response, predictor` arguments for the `roc` function. `formula, data` a formula (and possibly a data object) of type response~predictor for the `roc` function. `conf.level` the width of the confidence interval as [0,1], never in percent. Default: 0.95, resulting in a 95% CI. `boot.n` the number of bootstrap replicates. Default: 2000. `boot.stratified` should the bootstrap be stratified (default, same number of cases/controls in each replicate than in the original sample) or not. `thresholds` on which thresholds to evaluate the CI. Either the numeric values of the thresholds, a logical vector (as index of `roc\$thresholds`) or a character “all”, “local maximas” or “best” that will be used to determine the threshold(s) on the supplied curve with `coords` (not on the resampled curves). `progress` the name of progress bar to display. Typically “none”, “win”, “tk” or “text” (see the `name` argument to `create_progress_bar` for more information), but a list as returned by `create_progress_bar` is also accepted. See also the “Progress bars” section of this package's documentation. `parallel` if TRUE, the bootstrap is processed in parallel, using parallel backend provided by plyr (foreach). `...` further arguments passed to or from other methods, especially arguments for `roc` and `ci.thresholds.roc` when calling `ci.thresholds.default` or `ci.thresholds.formula`. Arguments for `txtProgressBar` (only `char` and `style`) if applicable. Arguments `best.method` and `best.weights` to `coords`.

### Details

`ci.thresholds.formula` and `ci.thresholds.default` are convenience methods that build the ROC curve (with the `roc` function) before calling `ci.thresholds.roc`. You can pass them arguments for both `roc` and `ci.thresholds.roc`. Simply use `ci.thresholds` that will dispatch to the correct method.

This function creates `boot.n` bootstrap replicate of the ROC curve, and evaluates the sensitivity and specificity at thresholds given by the `thresholds` argument. Then it computes the confidence interval as the percentiles given by `conf.level`.

A threshold given as a `logical` vector or `character` is converted to the corresponding numeric vector once using the supplied ROC curve, and not at each bootstrap iteration. See `ci.coords` for the latter behaviour.

For more details about the bootstrap, see the Bootstrap section in this package's documentation.

### Value

A list of length 2 and class “ci.thresholds”, “ci” and “list” (in this order), with the confidence intervals of the CI and the following items:

 `specificity` a matrix of CI for the specificity. Row (names) are the thresholds, the first column the lower bound, the 2nd column the median and the 3rd column the upper bound. `sensitivity` same than specificity.

Additionally, the list has the following attributes:

 `conf.level` the width of the CI, in fraction. `boot.n` the number of bootstrap replicates. `boot.stratified` whether or not the bootstrapping was stratified. `thresholds` the thresholds, as given in argument. `roc` the object of class “roc” that was used to compute the CI.

### Warnings

If `boot.stratified=FALSE` and the sample has a large imbalance between cases and controls, it could happen that one or more of the replicates contains no case or control observation, producing a `NA` area. The warning “NA value(s) produced during bootstrap were ignored.” will be issued and the observation will be ignored. If you have a large imbalance in your sample, it could be safer to keep `boot.stratified=TRUE`.

### References

James Carpenter and John Bithell (2000) “Bootstrap condence intervals: when, which, what? A practical guide for medical statisticians”. Statistics in Medicine 19, 1141–1164. DOI: \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F")}.

Tom Fawcett (2006) “An introduction to ROC analysis”. Pattern Recognition Letters 27, 861–874. DOI: \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1016/j.patrec.2005.10.010")}.

Xavier Robin, Natacha Turck, Alexandre Hainard, et al. (2011) “pROC: an open-source package for R and S+ to analyze and compare ROC curves”. BMC Bioinformatics, 7, 77. DOI: \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1186/1471-2105-12-77")}.

Hadley Wickham (2011) “The Split-Apply-Combine Strategy for Data Analysis”. Journal of Statistical Software, 40, 1–29. URL: \Sexpr[results=rd]{tools:::Rd_expr_doi("10.18637/jss.v040.i01")}.

`roc`, `ci`

### Examples

``````data(aSAH)

# Create a ROC curve:
data(aSAH)
roc1 <- roc(aSAH\$outcome, aSAH\$s100b)

## Basic example ##
# Compute CI of all local maxima thresholds
## Not run:
ci.thresholds(roc1)
## End(Not run)

## More options ##
# Customized bootstrap and thresholds:
## Not run:
ci.thresholds(roc1,
thresholds=c(0.5, 1, 2),
boot.n=10000, conf.level=0.9, stratified=FALSE)
## End(Not run)

## Plotting the CI ##
## Not run:
ci1 <- ci.thresholds(roc1)
## End(Not run)
plot(roc1)
plot(ci1)
``````

pROC documentation built on Nov. 2, 2023, 6:05 p.m.