boot_diff: Calculate the bootstrapped empiric two-sided p-value and...

In gweissman/gmish: Miscellaneous Functions for Predictive Modeling

Calculate the bootstrapped empiric two-sided p-value and difference between two sets of predictions for a given performance metric.

Description

Calculate the bootstrapped empiric two-sided p-value and difference between two sets of predictions for a given performance metric.

Usage

boot_diff(
  preds1,
  preds2,
  obs,
  metric = NULL,
  reps = 1000,
  conf = 0.95,
  seed = NULL,
  ...
)

Arguments

preds1	A vector of predicted probabilities for the first model.
preds2	A vector of predicted probabilities for the second model.
obs	A vector containing the observed binary outcomes (0 or 1).
metric	A function of the form f(preds, obs), e.g. 'bs'
reps	The number of bootstrap replicates. Default = 1000.
conf	The width of the confidence interval. Default = 0.95.
seed	An optional random seed.
...	Additional arguments for the particular metric function, e.g. 'thresh = 0.6'

Examples

# Generate some predictions for two different models
p1 <- runif(1000)
p2 <- runif(1000)
# Generate some binary outcomes
observations <- sample(0:1, size = 1000, replace = TRUE)
# Calculate the Confidence interval around the estimate of the Brier Score
boot_diff(p1, p2, observations, metric = brier)

gweissman/gmish documentation built on Feb. 21, 2025, 1:20 a.m.