boot_test: Test for equivalence of a metric

In cutpointr: Determine and Evaluate Optimal Cutpoints in Binary Classification Tasks

Test for equivalence of a metric

Description

This function performs a significance test based on the bootstrap results of cutpointr to test whether a chosen metric is equal between subgroups or between two cutpointr objects. The test statistic is calculated as the standardized difference of the metric between groups. If x contains subgroups, the test is run on all possible pairings of subgroups. An additional adjusted p-value is returned in that case.

Usage

boot_test(x, y = NULL, variable = "AUC", in_bag = TRUE, correction = "holm")

Arguments

x	A cutpointr object with bootstrap results
y	If x does not contain subgroups another cutpointr object
variable	The variable for testing
in_bag	Whether the in-bag or out-of-bag results should be used for testing
correction	The type of correction for multiple testing. Possible values are as in p.adjust.methods

Details

The variable name is looked up in the columns of the bootstrap results where the suffixes _b and _oob indicate in-bag and out-of-bag estimates, respectively (controlled via the in_bag argument). Possible values are optimal_cutpoint, AUC, acc, sensitivity, specificity, and the metric that was selected in cutpointr. Note that there is no "out-of-bag optimal cutpoint", so when selecting variable = optimal_cutpoint the test will be based on the in-bag data.

The test statistic is calculated as z = (t1 - t2) / sd(t1 - t2) where t1 and t2 are the metric values on the full sample and sd(t1 - t2) is the standard deviation of the differences of the metric values per bootstrap repetition. The test is two-sided.

If two cutpointr objects are compared and the numbers of bootstrap repetitions differ, the smaller number will be used.

Since pairwise differences are calculated for this test, the test function does not support multiple optimal cutpoints, because it is unclear how the differences should be calculated in that case.

Value

A data.frame (a tibble) with the columns test_var, p, d, sd_d, z and in_bag. If a grouped cutpointr object was tested, the additional columns subgroup1, subgroup2 and p_adj are returned.

Source

Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M. (2011). pROC: An open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12(1), 77. https://doi.org/10.1186/1471-2105-12-77

See Also

Other main cutpointr functions: add_metric(), boot_ci(), cutpointr(), multi_cutpointr(), predict.cutpointr(), roc()

Examples

## Not run: 
library(cutpointr)
library(dplyr)
set.seed(734)
cp_f <- cutpointr(suicide %>% filter(gender == "female"), dsi, suicide,
  boot_runs = 1000, boot_stratify = TRUE)
set.seed(928)
cp_m <- cutpointr(suicide %>% filter(gender == "male"), dsi, suicide,
  boot_runs = 1000, boot_stratify = TRUE)
# No significant differences:
boot_test(cp_f, cp_m, AUC, in_bag = TRUE)
boot_test(cp_f, cp_m, sum_sens_spec, in_bag = FALSE)

set.seed(135)
cp <- cutpointr(suicide, dsi, suicide, gender, boot_runs = 1000,
  boot_stratify = TRUE)
# Roughly same result as above:
boot_test(cp, variable = AUC, in_bag = TRUE)
boot_test(cp, variable = sum_sens_spec, in_bag = FALSE)

## End(Not run)

cutpointr documentation built on April 14, 2022, 1:06 a.m.