Nothing
#' Minimum sample size for custom simulation workflows
#'
#' Compute the minimum sample size required to achieve a target level of
#' predictive performance using user-defined simulation components.
#' `simulate_custom()` is the low-level interface in `pmsims`: users supply a
#' data-generating function, a model-fitting function, and a metric function,
#' and the chosen search engine estimates the smallest \eqn{n} meeting the
#' selected performance criterion.
#'
#' @param data_function Function taking a single argument, `n`, giving the
#' training sample size, and returning a dataset that can be passed to
#' `model_function`.
#' @param model_function Function that fits a model to the dataset returned by
#' `data_function`. It must take the generated dataset as its only argument
#' and return a fitted model object.
#' @param metric_function Function that evaluates predictive performance on test
#' data. It must take three positional arguments in the order
#' `(test_data, fitted_model, model_name)` and return a single numeric value.
#' Optionally, users may set `attr(metric_function, "value_on_error")` to a
#' single numeric fallback value to be returned if model fitting or metric
#' evaluation fails during a simulation run.
#' @param target_performance Numeric target value for the chosen performance
#' metric. The search aims to find the smallest sample size \eqn{n} for which
#' the selected criterion is met relative to this threshold.
#' @param c_statistic Optional numeric value used only by the internal
#' start-value heuristics for some outcome and metric combinations. In most
#' custom workflows this should be left as `NULL`.
#' @param mean_or_assurance Character string specifying the criterion used to
#' define the minimum sample size. Must be either `"mean"` or `"assurance"`.
#' @param test_n Integer size of the test dataset used to evaluate model
#' performance. This should usually be large enough that test-set variability
#' is negligible relative to the training-sample search.
#' @param min_sample_size Optional integer lower bound for the sample-size
#' search. If supplied, `max_sample_size` must also be supplied.
#' @param max_sample_size Optional integer upper bound for the sample-size
#' search. If supplied, `min_sample_size` must also be supplied.
#' Supplying both bounds defines the search space directly, so the adaptive
#' starting-value search is skipped. Because the runtime estimate is
#' extrapolated from that stage, no long-run warning is issued either.
#' @param n_reps_total Integer total number of simulation replications allocated
#' to the search. The search evaluates approximately
#' `n_reps_total / n_reps_per` candidate sample sizes.
#' @param n_reps_per Integer number of simulation replications performed at each
#' candidate sample size.
#' @param method Character string specifying the search engine: `"curve"`
#' (default; fits a learning curve to all replicates and searches where it
#' crosses the target, see Details), `"mlpwr"` (adaptive start values, then
#' mlpwr's Gaussian-process search), `"bisection"` or `"mlpwr-bs"`.
#' @param progress Logical flag controlling whether a progress bar is shown:
#' the curve engine's own, or mlpwr's for the `mlpwr`-based methods.
#' @param verbose Logical flag controlling engine-specific diagnostic output
#' when supported. For the bisection engine, setting `verbose = TRUE` stores
#' the iteration history on the returned object.
#' @param verify_reps Integer number of fresh replicates simulated at the
#' returned sample size to check that it meets the target (see `status`).
#' Set to `0` to skip the check.
#' @param max_n Largest sample size the search may try. If performance there is
#' still clearly below the target, the search stops with status
#' `"not_bracketed"`. Defaults to 200,000 for random forests and xgboost
#' (one batch of replicates at a million rows would take hours) and
#' 1,000,000 otherwise.
#' @param cores Number of cores for running the replicates of each batch in
#' parallel (Unix-like systems only). Results are the same for any number of
#' cores, because each replicate has its own random-number stream.
#' @param live_plot Logical; with `method = "curve"`, redraw the learning
#' curve after each batch of replicates so the search can be watched as it
#' runs (interactive sessions with a graphics device only; it does not
#' change the results).
#' @param ... Additional arguments passed to the selected search engine.
#'
#' @return An object of class `"pmsims"` containing the estimated minimum
#' sample size `min_n` (numeric; `NA` when no sample size could be found)
#' and a `status`:
#' \describe{
#' \item{`"ok"`}{The search found `min_n`, and simulating it again
#' confirmed the target is met (or the check was skipped).}
#' \item{`"not_verified"`}{Simulating `min_n` again gave performance
#' clearly below the target, so `min_n` is likely too small.}
#' \item{`"not_bracketed"`}{No sample size up to `max_n` met the target,
#' or the search returned no sample size.}
#' \item{`"unreachable"`}{With `method = "curve"`: the learning curve
#' levels off below the target, so no sample size reaches it.}
#' \item{`"replicates_failed"`}{At least half the simulation replicates
#' failed to fit or score the model, so no sample size was estimated.
#' With `method = "curve"`, also when 20% or more failed at every
#' sample size tried, so no learning curve could be fitted.}
#' }
#' `status_message` explains a status other than `"ok"`, `verification`
#' holds the check at `min_n`, and `diagnostics` records the search bounds,
#' whether `min_n` lies on one of them, mlpwr's warnings, Gaussian-process
#' restarts and failed replicates.
#'
#' @details
#' With `method = "curve"`, every replicate is kept and a monotone learning
#' curve \eqn{C(n) = a - b n^{-c}} is fitted to the criterion at each sample
#' size evaluated (weighted by a smoothed model of the replicate spread; the
#' ceiling \eqn{a} is capped at the metric's maximum). A pilot doubles
#' (or halves) from a heuristic start until the criterion is seen on both sides
#' of the target; each further batch of `n_reps_per` replicates is placed where
#' the fitted curve crosses the target. The search range is therefore never
#' fixed. The answer is where the final fitted curve crosses the target. A
#' target is declared unreachable if a bootstrap upper bound for the curve's
#' ceiling \eqn{a} is below it; the search also stops (status
#' `"not_bracketed"`) when the crossing is very likely beyond `max_n`. These
#' stops extrapolate the fitted curve and should be read as heuristics.
#'
#' The answer is a median-unbiased estimate of where the criterion reaches the
#' target. The
#' interval in `diagnostics$curve$n_ci` reflects Monte Carlo error given the
#' learning-curve shape, not uncertainty about the shape; a shape-free check
#' (`diagnostics$crosscheck_n`) is reported alongside, and flagged when it
#' differs by more than 10%. The verification of the answer has limited power
#' for skewed metrics such as CSSE: passing it does not show that the target
#' is met.
#'
#' @section Random numbers:
#' Each simulation replicate runs on its own random-number stream, derived from
#' a single draw from the session's RNG. Results are therefore reproducible
#' with [set.seed()], and a replicate's data do not depend on how many random
#' numbers other parts of the search consumed.
#'
#' @seealso [simulate_binary()], [simulate_continuous()], [simulate_survival()]
#'
#' @examples
#' # Three independent predictors with a population R-squared of 0.5.
#' data_fun <- function(n) {
#' x1 <- rnorm(n)
#' x2 <- rnorm(n)
#' x3 <- rnorm(n)
#' y <- (x1 + x2 + x3) / sqrt(3) + rnorm(n)
#' data.frame(y = y, x1 = x1, x2 = x2, x3 = x3)
#' }
#'
#' model_fun <- function(dat) {
#' stats::lm(y ~ ., data = dat)
#' }
#'
#' # Calibration slope evaluated on independent test data.
#' metric_fun <- function(test_data, fit, model) {
#' preds <- stats::predict(fit, newdata = test_data)
#' unname(stats::coef(stats::lm(test_data$y ~ preds))[2])
#' }
#' attr(metric_fun, "metric") <- "calibration_slope"
#'
#' \donttest{
#' set.seed(123)
#' est <- simulate_custom(
#' data_function = data_fun,
#' model_function = model_fun,
#' metric_function = metric_fun,
#' target_performance = 0.9,
#' mean_or_assurance = "assurance",
#' min_sample_size = 25,
#' max_sample_size = 1000,
#' n_reps_total = 1000,
#' test_n = 30000,
#' progress = FALSE
#' )
#' est
#' est$min_n
#' plot(est)
#' }
#' @export
simulate_custom <- function(
data_function,
model_function,
metric_function,
target_performance,
c_statistic = NULL,
mean_or_assurance = "assurance",
test_n = 30000,
min_sample_size = NULL,
max_sample_size = NULL,
n_reps_total = 1000,
n_reps_per = 20,
method = "curve",
progress = TRUE,
verbose = FALSE,
verify_reps = 100,
max_n = NULL,
cores = 1L,
live_plot = FALSE,
...
) {
# Evaluate four initial sample sizes after establishing the search bounds.
n_init <- 4
se_final <- NULL # Reserved for internal engine use.
if (is.null(data_function)) {
stop("data_function missing")
}
if (is.null(n_reps_total)) {
stop("'n_reps_total' must be specified.")
}
# Validate the optional sample-size bounds.
if (
(!is.null(min_sample_size) && is.null(max_sample_size)) ||
(is.null(min_sample_size) && !is.null(max_sample_size))
) {
stop(
"min_sample_size and max_sample_size must either both be positive integers or both set to NULL"
)
}
if (
!is.null(min_sample_size) &&
!is.null(max_sample_size) &&
min_sample_size > max_sample_size
) {
stop("min_sample_size must be less than max_sample_size")
}
if (!is.null(min_sample_size)) {
cli::cli_alert_info(
"Using user-specified min_sample_size and max_sample_size. \\
Adaptive starting values will not be used."
)
}
if ((mean_or_assurance %in% c("mean", "assurance")) == FALSE) {
stop("mean_or_assurance must be either 'mean' or 'assurance'")
}
check_metric_direction(attr(metric_function, "metric", exact = TRUE))
if (is.null(max_n)) {
max_n <- default_max_n(attr(model_function, "model", exact = TRUE))
}
# Choose the metric-specific fallback used when a simulation fails.
value_on_error <- resolve_value_on_error(metric_function)
time_1 <- Sys.time()
# One evaluator, on keyed random streams, for every stage of the search
# (see R/simulation_core.R).
# Replicates within each batch can run in parallel (forked processes, so
# not on Windows). Results do not depend on the number of cores: each
# replicate has its own random-number stream. Random forests and xgboost
# then use one thread each, so the workers do not compete for cores.
# `parallel = TRUE` without `cores`, as in earlier versions, uses up to 20.
if (missing(cores) && isTRUE(list(...)$parallel)) {
cores <- min(20L, parallel::detectCores(), na.rm = TRUE)
}
if (
!is.numeric(cores) ||
length(cores) != 1L ||
!is.finite(cores) ||
cores < 1 ||
cores != round(cores)
) {
cli::cli_abort("{.arg cores} must be a single positive whole number.")
}
cores <- as.integer(cores)
if (cores > 1L && .Platform$OS.type != "unix") {
cli::cli_alert_info(
"Parallel replicates need a Unix-like system; running on one core."
)
cores <- 1L
}
if (cores > 1L) {
old_threads <- options(pmsims.threads = 1L)
on.exit(options(old_threads), add = TRUE)
}
streams <- new_simulation_streams()
evaluator <- new_evaluator(
data_function = data_function,
model_function = model_function,
metric_function = metric_function,
test_n = test_n,
value_on_error = value_on_error,
streams = streams,
parallel = cores > 1L,
cores = cores
)
if (method == "curve") {
output <- do.call(
calculate_curve,
utils::modifyList(
list(
test_n = test_n,
n_reps_total = n_reps_total,
n_reps_per = n_reps_per,
se_final = se_final,
min_sample_size = min_sample_size,
max_sample_size = max_sample_size,
target_performance = target_performance,
c_statistic = c_statistic,
mean_or_assurance = mean_or_assurance,
progress = progress,
data_function = data_function,
model_function = model_function,
metric_function = metric_function,
value_on_error = value_on_error,
evaluator = evaluator,
max_n = max_n,
live_plot = live_plot
),
list(...)
)
)
} else if (method == "mlpwr") {
output <- do.call(
calculate_mlpwr,
utils::modifyList(
list(
test_n = test_n,
n_reps_total = n_reps_total,
n_reps_per = n_reps_per,
se_final = se_final,
min_sample_size = min_sample_size,
max_sample_size = max_sample_size,
target_performance = target_performance,
c_statistic = c_statistic,
mean_or_assurance = mean_or_assurance,
n_init = n_init,
progress = progress,
verbose = verbose,
data_function = data_function,
model_function = model_function,
metric_function = metric_function,
value_on_error = value_on_error,
evaluator = evaluator,
max_n = max_n
),
list(...)
)
)
} else if (method == "bisection") {
output <- do.call(
calculate_bisection,
utils::modifyList(
list(
data_function = data_function,
model_function = model_function,
metric_function = metric_function,
value_on_error = value_on_error,
min_sample_size = min_sample_size,
max_sample_size = max_sample_size,
test_n = test_n,
n_reps_total = n_reps_total,
n_reps_per = n_reps_per,
target_performance = target_performance,
c_statistic = c_statistic,
mean_or_assurance = mean_or_assurance,
tol = 1e-3,
verbose = verbose,
budget = TRUE,
evaluator = evaluator,
max_n = max_n
),
list(...)
)
)
} else if (method == "mlpwr-bs") {
output <- do.call(
calculate_mlpwr_bs,
utils::modifyList(
list(
test_n = test_n,
n_reps_total = n_reps_total,
n_reps_per = n_reps_per,
se_final = se_final,
min_sample_size = min_sample_size,
max_sample_size = max_sample_size,
target_performance = target_performance,
c_statistic = c_statistic,
mean_or_assurance = mean_or_assurance,
progress = progress,
verbose = verbose,
data_function = data_function,
model_function = model_function,
metric_function = metric_function,
value_on_error = value_on_error,
evaluator = evaluator,
max_n = max_n
),
list(...)
)
)
} else {
stop("Method not found")
}
check <- check_result(
output = output,
evaluator = evaluator,
target_performance = target_performance,
mean_or_assurance = mean_or_assurance,
verify_reps = verify_reps,
describe = describe_value(metric_function),
quantile_type = if (method == "curve") 8L else 7L
)
time_2 <- Sys.time()
results_list <- list(
outcome = attr(data_function, "outcome"),
min_n = check$min_n,
perf_n = check$perf_n,
status = check$status,
status_message = check$status_message,
verification = check$verification,
diagnostics = check$diagnostics,
mlpwr_ds = output$mlpwr_ds,
target_performance = target_performance,
summaries = output$summaries,
data = output$results,
train_size = rownames(output$results),
data_function = data_function,
model_function = model_function,
metric_function = metric_function,
model = attr(model_function, "model", exact = TRUE),
metric = attr(metric_function, "metric", exact = TRUE),
c_statistic = c_statistic,
test_n = test_n,
min_sample_size = min_sample_size,
max_sample_size = max_sample_size,
n_reps_total = n_reps_total,
n_reps_per = n_reps_per,
method = method,
progress = progress,
verbose = verbose,
simulation_time = difftime(time_2, time_1, units = "secs"),
# Searches restarted after a Gaussian-process surrogate failure (mlpwr engines).
gp_restarts = output$gp_restarts %||% 0L,
rng_base_seed = streams$base_seed,
# So that follow-up simulations (e.g. the wrappers' secondary metric)
# can use the same number of cores.
cores = cores,
mean_or_assurance = mean_or_assurance
)
if (!is.null(output$history)) {
results_list$history <- output$history
}
attr(results_list, "class") <- "pmsims"
return(results_list)
}
# Default largest sample size: lower for random forests and xgboost, where one
# batch of replicates at a million rows would take hours.
default_max_n <- function(model) {
if (length(model) == 1L && !is.na(model) && model %in% c("rf", "xgboost")) {
2e5
} else {
1e6
}
}
resolve_value_on_error <- function(metric_function) {
metric_name <- attr(metric_function, "metric", exact = TRUE)
custom_value_on_error <- attr(metric_function, "value_on_error", exact = TRUE)
error_values <- c(
auc = 0.5,
cindex = 0.5,
r2 = 0,
brier_score_scaled = 0,
brier_score = 1,
ibs = 1,
calibration_slope = 0,
# CSSE of a calibration slope of 0, matching calibration_slope above. CSSE
# is <= 0 with 0 perfect, so the generic 0.5 fallback would score a failed
# fit as better than perfect calibration.
csse = -1
)
if (!is.null(custom_value_on_error)) {
if (
!is.numeric(custom_value_on_error) ||
length(custom_value_on_error) != 1 ||
is.na(custom_value_on_error)
) {
stop(
"attr(metric_function, \"value_on_error\") must be a single non-missing numeric value."
)
}
return(as.numeric(custom_value_on_error))
}
if (
length(metric_name) == 1 &&
!is.na(metric_name) &&
metric_name %in% names(error_values)
) {
return(unname(error_values[[metric_name]]))
}
cli::cli_alert_info(paste(
"Failed replicates will count as a performance of 0.5. Set",
"{.code attr(metric_function, \"value_on_error\")} to choose another value."
))
0.5
}
# Fallback for a metric value that is not a single finite number. Metrics can
# fail without raising an error, e.g. a lasso that selects no predictors gives
# constant predictions and an NA calibration slope. Those replicates must count
# as failures, like errors do; otherwise na.rm = TRUE in the summaries drops
# them and the remaining replicates overstate performance.
metric_or_fallback <- function(value, value_on_error) {
if (is.numeric(value) && length(value) == 1L && is.finite(value)) {
value
} else {
value_on_error
}
}
#' Check an engine's answer and assign the result status
#'
#' Engines can stop early (the adaptive stage reached max_n without meeting
#' the target). Otherwise the returned sample size is simulated again
#' with fresh replicates: mlpwr's reported performance is its surrogate's
#' prediction, never an observed value, so without this check an answer for an
#' unreachable target -- for example the edge of the search range -- is
#' returned as if it met the target.
#' @keywords internal
#' @noRd
check_result <- function(
output,
evaluator,
target_performance,
mean_or_assurance,
verify_reps = 100,
describe = function(x) format(signif(x, 4)),
quantile_type = 7L
) {
search <- output$search %||% list()
min_n <- suppressWarnings(as.numeric(output$min_n))
perf_n <- suppressWarnings(as.numeric(output$perf_n))
if (length(min_n) != 1L) {
min_n <- NA_real_
}
if (length(perf_n) != 1L) {
perf_n <- NA_real_
}
status <- search$status
status_message <- search$status_message
verification <- NULL
# Mostly failed replicates make any answer meaningless, whatever the search
# returned.
failed <- failure_status(evaluator)
if (!is.null(failed)) {
status <- failed$status
status_message <- failed$message
}
if (is.null(status)) {
if (!is.finite(min_n)) {
status <- "not_bracketed"
status_message <- paste(
"The search did not return a sample size: no sample size in the",
"search range was predicted to meet the target."
)
} else if (isTRUE(verify_reps > 0)) {
verification <- verify_sample_size(
evaluator = evaluator,
n = min_n,
target_performance = target_performance,
mean_or_assurance = mean_or_assurance,
reps = verify_reps,
type = quantile_type
)
if (verification$verified) {
status <- "ok"
} else {
status <- "not_verified"
status_message <- sprintf(
paste(
"Simulating n = %s again gave %s, clearly below the target %s.",
"The target may be unreachable, or the search range may not",
"contain the answer; this sample size is likely too small."
),
format(min_n, big.mark = ",", scientific = FALSE),
describe(verification$performance),
describe(target_performance)
)
}
} else {
status <- "ok"
}
}
if (!status %in% c("ok", "not_verified")) {
min_n <- perf_n <- NA_real_
}
if (!identical(status, "ok")) {
warning(status_message, call. = FALSE)
} else if (identical(search$at_bound, "lower")) {
cli::cli_alert_info(paste(
"The estimate lies on the lower edge of the search range",
"({search$bounds[1]}-{search$bounds[2]}), so it may overstate the",
"sample size needed."
))
} else if (identical(search$at_bound, "upper")) {
cli::cli_alert_info(paste(
"The estimate lies on the upper edge of the search range",
"({search$bounds[1]}-{search$bounds[2]}); the sample size needed may",
"be larger."
))
}
if (identical(status, "ok")) {
ci_text <- function(ci) {
sprintf(
"%s to %s",
round(ci[1]),
if (is.finite(ci[2])) round(ci[2]) else "Inf"
)
}
if (isTRUE(search$near_ceiling)) {
cli::cli_alert_warning(paste(
"The target is close to the best performance this model can reach:",
"doubling the sample size from the answer gains less than the Monte",
"Carlo error, so the sample size is poorly determined."
))
} else if (isTRUE(search$poorly_determined)) {
cli::cli_alert_warning(paste(
"The sample size is poorly determined: its interval is",
"{ci_text(search$curve$n_ci)}."
))
}
if (isTRUE(search$crosscheck_disagrees)) {
cli::cli_alert_warning(paste(
"A shape-free check (a monotone fit to the simulated points) puts the",
"answer at {round(search$crosscheck_n)}, more than 10% from the",
"learning-curve answer."
))
}
}
if (length(search$mlpwr_warnings)) {
cli::cli_alert_warning(
"mlpwr reported: {paste(search$mlpwr_warnings, collapse = '; ')}"
)
}
failures <- evaluator$failures()
list(
min_n = min_n,
perf_n = perf_n,
status = status,
status_message = status_message %||% NA_character_,
verification = verification,
diagnostics = list(
bounds = search$bounds,
at_bound = search$at_bound %||% NA_character_,
adaptive_stop_reason = search$adaptive_stop_reason %||% NA_character_,
max_achievable_perf = search$max_achievable_perf %||% NA_real_,
mlpwr_warnings = search$mlpwr_warnings %||% character(0),
gp_restarts = output$gp_restarts %||% 0L,
curve = search$curve,
near_ceiling = isTRUE(search$near_ceiling),
poorly_determined = isTRUE(search$poorly_determined),
gain_per_doubling = search$gain_per_doubling %||% NA_real_,
crosscheck_n = search$crosscheck_n %||% NA_real_,
crosscheck_disagrees = isTRUE(search$crosscheck_disagrees),
shape_at_bound = search$shape_at_bound %||% NA_character_,
replicates = failures,
failed_replicates = sum(failures$failed),
total_replicates = sum(failures$reps)
)
)
}
#' Parse and validate input specifications
#'
#' This function validates the provided data, model, and metric specifications,
#' and returns corresponding generator functions for each. It ensures that all
#' required inputs are provided and correctly configured.
#'
#' @param data_spec A list containing two elements:
#' \describe{
#' \item{\code{type}}{A character string indicating the outcome type.}
#' \item{\code{args}}{A list of arguments to be passed to the data-generating function.}
#' }
#' @param metric A character vector specifying one or more metrics to be used.
#' Currently, only the first element is used.
#' @param model A character string specifying the model to be used.
#'
#' @return A list containing three elements:
#' \describe{
#' \item{\code{data_function}}{The data-generating function.}
#' \item{\code{model_function}}{The model-generating function.}
#' \item{\code{metric_function}}{The metric function corresponding to the chosen metric.}
#' }
#'
#' @details
#' This function calls \code{default_data_generators()}, \code{default_model_generators()},
#' and \code{default_metric_generator()} to construct the appropriate functions based on
#' the supplied inputs.
#'
#' @keywords internal
parse_inputs <- function(data_spec, metric, model) {
if (is.null(metric)) {
stop("metric is missing")
}
if (is.null(data_spec)) {
stop("data_spec missing")
}
data_function <- default_data_generators(data_spec)
model_function <- default_model_generators(
attr(data_function, "outcome"),
model
)
# The current interface uses the first requested metric.
metric_function <- default_metric_generator(metric[[1]], data_function)
return(list(
data_function = data_function,
model_function = model_function,
metric_function = metric_function
))
}
Any scripts or data that you put into this service are public.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.