Bootstrap paired ROC curves

Description

Given two numerical predictors for the same outcome on the same set of samples, this functions enables the bootstrapping of the paired ROC curves of the two prediction models. While bootstrapping the same set of samples are used for both curves in each iteration, preserving the correlation between the two models.

Usage

1
2
boot.paired.roc(pred1, pred2, true.class, stratify = TRUE, n.boot = 1000,
  use.cache = FALSE, tie.strategy = NULL)

Arguments

pred1

Numerical predictions for the first classifier.

pred2

Numerical predictions for the second classifier.

true.class

A logical vector. TRUE indicates the sample belonging to the positive class.

stratify

Logical. Indicates whether we use stratified bootstrap. Default to TRUE. Non-stratified bootstrap is not yet implemented.

n.boot

A number that will be coerced to integer. Specified the number of bootstrap replicates. Defaults to 1000.

use.cache

If true the bootstrapping results for the ROC curve will be pre-cached. This increases speed when the object is used often, but also takes up more memory.

tie.strategy

How to handle ties. See details below.

Value

A list of class fbroc.paired.roc, containing the elements:

prediction1

Input predictions for first model.

prediction2

Input predictions for second model.

true.class

Input classes.

n.thresholds1

Number of thresholds of the first predictor.

n.thresholds2

Number of thresholds of the second predictor.

n.boot

Number of bootstrap replicates.

use.cache

Indicates if cache is used for this ROC object.

tie.strategy

Used setting how to handle ties in predictors.

n.pos

Number of positive observations.

n.neg

Number of negative observations.

roc1

A data.frame containing the thresholds of the first ROC curve and the TPR and FPR at these thresholds.

roc2

A data.frame containing the thresholds of the second ROC curve and the TPR and FPR at these thresholds.

auc1

The AUC of the first ROC curve.

auc2

The AUC of the second ROC curve.

boot.tpr1

If the cache is enabled, a matrix containing the bootstrapped TPR at the thresholds for the first predictor.

boot.fpr1

If the cache is enabled, a matrix containing the bootstrapped FPR at the thresholds for the first predictor.

boot.tpr2

If the cache is enabled, a matrix containing the bootstrapped TPR at the thresholds for the second predictor.

boot.fpr2

If the cache is enabled, a matrix containing the bootstrapped FPR at the thresholds for the second predictor.

Caching

If you enable caching, boot.roc calculates the requested number of bootstrap samples and saves the TPR and FPR values for each iteration. This can take up a sizable portion of memory, but it speeds up subsequent operations. This can be useful if you plan to use the ROC curve multiple fbroc functions.

Ties

You can set this parameter to either 1 or 2. If your numerical predictor has no ties, both settings will produce the same results. If you set tie.strategy to 1 the ROC curve is built by connecting the TPR/FPR pairs for neighboring thresholds. A tie.strategy of 2 indicates that the TPR calculated at a specific FPR is the best TPR at a FPR smaller than or equal than the FPR specified. Defaults to 2.

See Also

boot.roc, plot.fbroc.paired.roc, perf.fbroc.paired.roc

Examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
data(roc.examples)
# Do not use cache
example <- boot.paired.roc(roc.examples$Cont.Pred, roc.examples$Cont.Pred.Outlier,
                          roc.examples$True.Class, n.boot = 500)
perf(example, "auc") # estimate difference in auc
perf(example, "tpr", fpr = 0.5) # estimate difference in TPR at a FPR of 50%
plot(example) # show plot
# Cached mode
example <- boot.paired.roc(roc.examples$Cont.Pred, roc.examples$Cont.Pred.Outlier,
                          roc.examples$True.Class, n.boot = 1000, use.cache = TRUE)
conf(example, conf.for = "tpr", steps = 10) # get confidence regions for TPR at FPR
conf(example, conf.for = "fpr", steps = 10) # get confidence regions for FPR at TPR
perf(example, "fpr", tpr = 0.9) # estimate difference in FPR at a TPR of 90%                     

Want to suggest features or report bugs for rdrr.io? Use the GitHub issue tracker.