pr.test: Compare two Precision-Recall curves
In bblodfon/usefun: A Collection of Useful Functions by John

pr.test

R Documentation

Compare two Precision-Recall curves

Description

Test the hypothesis that the true difference in PR AUCs is equal to 0. We implement the same bootstrap method based on the idea from pROC::roc.test(). The PR AUC is calculated using PRROC::pr.curve() with the interpolation method of Davis (2006).

Usage

pr.test(
  labels,
  pred1,
  pred2,
  boot.n = 10000,
  boot.stratified = TRUE,
  alternative = "two.sided"
)

Arguments

`labels`	`numeric()` Vector of responses/labels (only two classes/values allowed: cases/positive class = 1 and controls/negative class = 0)
`pred1`	`numeric()` Vector of prediction values. Higher values denote positive class.
`pred2`	`numeric()` Vector of prediction values. Higher values denote positive class. Must have the same length as `pred1`.
`boot.n`	`numeric(1)` Number of bootstrap resamples. Default: 10000
`boot.stratified`	`logical(1)` Whether the bootstrap resampling is stratified (same number of cases/controls in each replicate as in the original sample) or not. It is advised to use stratified resampling when classes from `labels` are imbalanced. Default: TRUE.
`alternative`	`character(1)` Specifies the alternative hypothesis. Either "two.sided", "less" or "greater". Default: "two.sided".

Value

a list with the AUCs of the two original prediction vectors and the p-value of the bootstrap-based test.

References

Davis J, Goadrich M (2006). “The relationship between precision-recall and ROC curves.” Proceedings of the 23rd International Conference on Machine Learning, 148(4), 233–240. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1145/1143844.1143874")}.

Examples

set.seed(42)
# imbalanced labels
labels = sample(c(0,1), 20, replace = TRUE, prob = c(0.8,0.2))
# predictions
pred1 = rnorm(20)
pred2 = rnorm(20)
pr.test(labels, pred1, pred2, boot.n = 1000, boot.stratified = FALSE)
pr.test(labels, pred1, pred2, boot.n = 1000, boot.stratified = TRUE)

bblodfon/usefun documentation built on April 29, 2024, 12:36 p.m.