study_mle: Select models based on validation performance and conduct an...

In maxwestphal/SEPM.SIM: Statistical Evaluation of Prediction Models - Simulation Studies

Select models based on validation performance and conduct an evaluation study

Description

This function reads in data instances produces via sample_mle() and emulates the process of conducting an evaluation study for one or multiple selected prediction models.

Usage

study_mle(
  instance,
  methods = NA,
  M = 200,
  M.start = NA,
  M.probs = c("uniform", "learn"),
  M.seed = 1,
  n.eval = 200,
  first.eval = 1,
  rdm.eval = FALSE,
  analysis = c("acc", "cpe"),
  delta = 0,
  shift = 0.05,
  select.method = c("close", "best", "optimal", "oracle", "simplest.en"),
  select.limit = c("none", "sqrt", "one"),
  select.args = "",
  estimate.method = "beta.approx",
  estimate.args = "",
  infer.method = "maxT",
  alternative = "greater",
  alpha = 0.025,
  transform = "none",
  data = NULL,
  job = NULL
)

Arguments

instance	simulation instance generated by sample_mle
methods	character, potentially subset available prediction models by method (=learning algorithm) e.g. recover elastic net models by specifying methods="glmnet" (caret train.method), no effect if methods=NA (default)
M	integer, number of models to subsample from available models (restricted via methods argument), needs to be less or euqal than number of available models (200 per default)
M.start	integer, starting index for subsetting
M.probs	character, "uniform" for random subset, "learn.theta" for P(selected)=learn.theta(=true model performance), "learn.theta.neg" for P(selected)=1-learn.theta
M.seed	integer, seed for random subsetting (i.e. if M.probs != "uniform")
n.eval	integer, test (evaluation) sample size
first.eval	integer, index of first evaluation observation (from all available)
rdm.eval	logical, choose test samples randomly? (default: FALSE)
analysis	character, either "acc" or "cpe"
delta	numeric (default: 0)
shift	numeric (default: 0.05)
select.method	character, selection method based on validation ranking, e.g. "rank" (default) or "se"
select.limit	integer, maximum number of models to evaluate
select.args	character, further arguments defining selection rule e.g. "r=1" for select.method="rank" to choose only best validation models or "c=1" for select.method="se" (which defines the 'within1SE# rule)
estimate.method	character, estimation method in SEPM package default ("beta.approx")
estimate.args	character, specify additional estimation argument as character of form "arg1=value1_arg2=value2_..."
infer.method	character, defines the statistical test, e.g. "maxT", "Bonferroni" or "naive"
alternative	character, either "greater" (default), "lower" or "two.sided"
alpha	numeric, significance level (default: 0.025)
transform	character, specifies transformation of test statistics, passed to SEPM::infer
data	ignored (required for batchtools compatibility)
job	ignored (required for batchtools compatibility)

Value

Returns a list which contains all relevant characteristics of the evaluation study.

maxwestphal/SEPM.SIM documentation built on April 11, 2024, 4:06 p.m.