Description Important fbroc functions Example Data Details Notes References Examples
Fbroc enables the fast bootstrap analysis and comparison of ROC curves for simulation studies and shiny applications by using a fast algorithm where the cost of a single bootstrap replicate is O(n), with n denoting the number of observations. The algorithm is implemented in C++ to further increase the efficiency. On a typical desktop computer the time needed for the calculation of 100000 bootstrap replicates given 500 observations requires time on the order of magnitude of one second. The ROC curve as used shows the True Positive Rate (TPR) as a function of the False Positive Rate (FPR). The package also support the analysis of paired ROC curves, where we compare two predictions given for the same set of samples.
boot.roc
Use boot.roc
to bootstrap a ROC curve.
boot.paired.roc
Use boot.paired.roc
to bootstrap two paired ROC curves.
conf
Calculate confidence regions for the ROC curve.
perf
Estimate performance and calculate confidence intervals.
fbroc also contains the example data set roc.examples, which you can use to test the functionality of the package. This data set contains simulated data and not an real application.
The algorithm works by first determining the critical thresholds of the ROC
curve - cutoffs at which the curve changes directions. Each observation is then linked
to the specific thresholds at which they first cause a change in the TPR
or FPR. Calculating this link and directly bootstrapping that link
allows us to construct the bootstrapped ROC
curve very quickly. Since multiple observation can be linked to the same
threshold, it is difficult to implement the algorithm efficiently in R.
This is why fbroc
implements it in C++.
When bootstrapping paired ROC curves, the packages takes care of using the same set of samples
for both predictors in each iteration of the bootstrap. This preserves the correlation structure
between both predictors.
All bootstrap confidence interval are based on the percentile method.
Package fbroc
is still in an early development stage. Currently it supports bootstrapping
the confidence region of single and paired ROC curves, as well as the AUC, partial AUC,
the FPR at a fixed TPR and vice versa.
More sophisticated bootstrap confidence interval
calculation and improved documentation will be added at a later time.
Efron, B., & Tibshirani, R. (1998). An introduction to the bootstrap. Boca Raton, Fla: Chapman & Hall/CRC.
Donna Katzman McClish. (1989). Analyzing a Portion of the ROC Curve. Medical Decision Making, http://mdm.sagepub.com/content/9/3/190.abstract.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | data(roc.examples)
# work with a single ROC curves
result.boot <- boot.roc(roc.examples$Cont.Pred, roc.examples$True.Class, n.boot = 100)
plot(result.boot)
perf(result.boot, "auc")
perf(result.boot, "auc", conf.level = 0.99)
perf(result.boot, "tpr", conf.level = 0.95, fpr = 0.1)
conf(result.boot, steps = 10)
# work with paired ROC curves
result.boot <- boot.paired.roc(roc.examples$Cont.Pred, roc.examples$Cont.Pred.Outlier,
roc.examples$True.Class, n.boot = 100)
plot(result.boot)
perf(result.boot, "auc")
perf(result.boot, "auc", conf.level = 0.99)
perf(result.boot, "tpr", conf.level = 0.95, fpr = 0.1)
conf(result.boot, steps = 10)
|
Loading required package: ggplot2
Warning: Ignoring unknown aesthetics: y
Bootstrapped ROC performance metric
Metric: AUC
Bootstrap replicates: 100
Observed: 0.929
Std. Error: 0.02
95% confidence interval:
0.881 0.959
Bootstrapped ROC performance metric
Metric: AUC
Bootstrap replicates: 100
Observed: 0.929
Std. Error: 0.019
99% confidence interval:
0.873 0.968
Bootstrapped ROC performance metric
Metric: TPR at a fixed FPR of 0.1
Bootstrap replicates: 100
Observed: 0.775
Std. Error: 0.081
95% confidence interval:
0.631 0.9
FPR TPR Lower.TPR Upper.TPR
1 1.0 1.0000 1.0000000 1.0000000
2 0.9 1.0000 1.0000000 1.0000000
3 0.8 1.0000 1.0000000 1.0000000
4 0.7 1.0000 1.0000000 1.0000000
5 0.6 1.0000 0.9875000 1.0000000
6 0.5 0.9875 0.9750000 1.0000000
7 0.4 0.9875 0.9375000 1.0000000
8 0.3 0.9375 0.8434375 1.0000000
9 0.2 0.8750 0.7934375 0.9500000
10 0.1 0.7750 0.5678125 0.8940625
11 0.0 0.4000 0.3125000 0.6315625
Warning: Ignoring unknown aesthetics: y
Warning: Ignoring unknown aesthetics: y
Bootstrapped ROC performance metric
Metric: AUC
Bootstrap replicates: 100
Classifier 1:
Observed:0.929
Std. Error: 0.021
95% confidence interval:
0.887 0.963
Classifier 2:
Observed:0.895
Std. Error: 0.031
95% confidence interval:
0.838 0.947
Delta:
Observed:0.034
Std. Error: 0.019
95% confidence interval:
0 0.083
Correlation: 0.79
Bootstrapped ROC performance metric
Metric: AUC
Bootstrap replicates: 100
Classifier 1:
Observed:0.929
Std. Error: 0.018
99% confidence interval:
0.883 0.962
Classifier 2:
Observed:0.895
Std. Error: 0.024
99% confidence interval:
0.81 0.932
Delta:
Observed:0.034
Std. Error: 0.018
99% confidence interval:
0.009 0.085
Correlation: 0.66
Bootstrapped ROC performance metric
Metric: TPR at a fixed FPR of 0.1
Bootstrap replicates: 100
Classifier 1:
Observed:0.775
Std. Error: 0.086
95% confidence interval:
0.561 0.919
Classifier 2:
Observed:0.675
Std. Error: 0.163
95% confidence interval:
0.355 0.889
Delta:
Observed:0.1
Std. Error: 0.139
95% confidence interval:
0 0.433
Correlation: 0.52
FPR Delta.TPR Lower.Delta.TPR Upper.Delta.TPR
1 1.0 0.0000 0 0.0000000
2 0.9 0.0000 0 0.0000000
3 0.8 0.0000 0 0.0000000
4 0.7 0.0000 0 0.0000000
5 0.6 0.0000 0 0.0125000
6 0.5 0.0000 0 0.0125000
7 0.4 0.0000 0 0.0625000
8 0.3 0.0375 0 0.0815625
9 0.2 0.0000 0 0.0940625
10 0.1 0.1000 0 0.4268750
11 0.0 0.4000 0 0.5750000
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.