bootLadderBoot: Calculate a bootstrapped score for an initial submission or...

In Sage-Bionetworks/challengescoring: Reducing Overfitting in Challenges With the BootLadderBoot Approach

Description

Calculate a bootstrapped score for an initial submission or subsequent submission.

Usage

bootLadderBoot(
  predictions,
  predictionColname,
  goldStandard,
  goldStandardColname,
  prevPredictions = NULL,
  scoreFun = spearman,
  bootstrapN = 10000,
  reportBootstrapN = 10,
  bayesThreshold = 3,
  seed = 98121,
  largerIsBetter = TRUE,
  verbose = FALSE,
  doParallel = FALSE
)

Arguments

predictions	The relative path to the current prediction csv, or a data frame.
predictionColname	The name of the column in the prediction csv that contains numeric prediction values. If also using a previous prediction file, must be the same name.
goldStandard	The relative path to the gold standard/test data csv, or a data frame.
goldStandardColname	The name of the column in the gold standard csv that contains numeric prediction values.
prevPredictions	If a previous prediction file for this team/participant already exists, pass in the path or data frame here. Prediction colname must match.
scoreFun	A scoring function. Default is Spearman correlation. Any function can be passed as long as it can calculate a score from two vectors (gold standard first, and prediction values second).
bootstrapN	Number of total bootstraps to perform (default 10000).
reportBootstrapN	Number of bootstraps to base returned score off of (default 10). The greater this value, the more accurate of a result is returned (and possibly the more the test data can be overfit).
bayesThreshold	The threshold for reporting a score. If the bayes factor (K) of the new prediction relative to the previous submission is less than this value, no score is returned. Default 3.
seed	Set a seed for bootstrap sampling. Default 98121.
largerIsBetter	Set this to FALSE if a smaller scoring metric indicates better performance (e.g. root mean squared error). Default TRUE.
verbose	Report step. Default FALSE.
doParallel	Bootstrap in parallel. Only works on UNIX based OS. Default TRUE.

Value

A named list with a bootstrapped score and a boolean stating whether the bayesThreshold was met. If verbose == T, also returns the calculated Bayes factor.

Sage-Bionetworks/challengescoring documentation built on Sept. 28, 2020, 8:27 a.m.