Nothing
subsample below 1 (the default is
0.8) can differ slightly with the number of threads: between cores = 1
and cores > 1, and with option pmsims.threads. xgboost's parallel row
sampling assumes a fixed number of random draws per row, which that library
does not guarantee. Results on other systems are unchanged. The test of
thread-invariance now uses subsample = 1, which fixes the check error on
CRAN's Fedora machines.method = "curve", is the default in
simulate_custom() and the simulate_*() wrappers; the previous engine is
available as method = "mlpwr". It fits a monotone learning curve to every
replicate simulated so far and puts each new batch where the curve crosses
the target, so there is no fixed search range. The answer is a
median-unbiased estimate with an interval
(diagnostics$curve$n_ci), checked against a shape-free fit
(diagnostics$crosscheck_n); targets close to the best achievable
performance are flagged. Targets that cannot be reached stop with status
"unreachable" and a message saying what is achievable; when 20% or more
of replicates fail at every sample size tried, so that no curve can be
fitted, it stops with status "replicates_failed".plot() shows the learning curve, the target and the answer for
method = "curve"; live_plot = TRUE redraws it during the search.cores runs the replicates of each batch in parallel; results are the same
for any number of cores.method = "curve", when performance levels off within Monte Carlo
error of the target, the search stops with status "not_bracketed" and a
message giving the performance at both ends of the last doubling, rather
than simulating ever larger samples.set.seed(), identical in serial and parallel, and
unaffected by how much of the search came before.verify_reps) and the result reports a status
("ok", "not_verified", "not_bracketed"). min_n is NA when no
answer was found.max_n itself as its last rung (it could stop
at about half of it)."replicates_failed" and no sample size.parallel = TRUE without cores uses up to 20 cores, as before. Parallel
replicates use forked processes, so they run serially on Windows; random
forests and xgboost then use one thread per worker.method = "mlpwr", n_reps_total is raised to 2000 for "lm",
"glm" and "coxph" models with at most 10 signal parameters, where the
Gaussian-process search is limited by its budget. A message says so, and
auto_n_reps_total = FALSE keeps the value supplied.max_n is the largest sample size the search may try, for every engine.
With method = "curve" it caps the whole search (unless max_sample_size
is given); with the mlpwr and bisection engines it caps the start-value
search, which sets their search range. It defaults to 200,000 for random
forests and xgboost and 1,000,000 otherwise.min_n and perf_n are NA when no sample size is found (they were a
character message); see status and status_message.simulate_*() wrappers no longer suppress warnings: a result that is
not "ok" raises a warning with its status_message, and mlpwr's own
warnings are reported.metric_2_at_n is the mean over 10 replicates at min_n (it was one).metric attribute count failed
replicates as 0.5 (with a message); set attr(metric_function,
"value_on_error") to choose another value.calibration_in_the_large, brier_score,
ibs) are rejected, since the search assumes larger values are better.survival_auc computes the time-dependent AUC (cumulative/dynamic, with
inverse-probability-of-censoring weights) itself rather than through
timeROC, which needs the survival package attached and otherwise left
survival_auc returning the concordance index. Results for survival AUC
targets change. timeROC is now in Suggests.pmsims.threads; xgboost previously used every core.correlation (the default is 0.3) will change.NA on test sets of more than about
90,000 rows (integer overflow).ridge continues its lambda path below glmnet's default end when
cross-validation selects the smallest lambda, so its calibration improves
with sample size. Results for ridge change, most at large sample sizes.lasso and ridge pass cox.ties = "breslow" to glmnet. This is
glmnet's current default, so results are unchanged, but glmnet 5.0 and later
warn on every Cox fit that does not set it, and plan to change the default
to Efron. glmnet 5.0 or later is now required.coxph uses the Cox calibration slope,
coxph(Surv ~ lp), with metric = "calibration_slope", as it already did
with metric = "csse" and as lasso and ridge do. rf and xgboost use
the calibration slope at the median event time. Results for coxph
calibration slope targets change.lm run with 80 predictors).complexity and data_control."csse") for all supported outcome
types, defined as -(1 - slope)^2 so that larger is better and 0 is perfect
calibration. Unlike the calibration slope itself, it penalises slopes above
one as well as below it.n. The
calibration slope does so for classical regression models, which overfit and
approach one from below, but not for penalised and tree-based learners, whose
slope can approach one from either side depending on how much the fit is
shrunk, and so can meet the target at more than one sample size. CSSE is a
squared distance from perfect calibration, so it improves monotonically
regardless of the direction of miscalibration. Requesting
metric = "calibration_slope" with lasso, ridge, rf, or xgboost now
searches on the CSSE scale internally and translates the answer back, so
targets and results stay on the calibration slope scale.simulate_custom()
and the wrapper functions.options(pmsims.confirm_long_runs = FALSE).Inputs is now divided into
Data-generating scenario, Model and performance, and Simulation, and
Results leads with the minimum sample
size, followed by the performance expected at that sample size.Data-generating scenario:
signal complexity, nonlinear strength, predictor distribution, predictor
correlation, and binary predictor prevalence.Performance at N = ... heading,
with the achieved value for the chosen metric shown alongside its target.Calibration slope >= 0.900, using the direction in which the metric
improves.Random forest rather than rf) and standardised names for statistical
quantities (AUC and the C-statistic are both reported as C-statistic for a
binary outcome).Results.print(x, verbose = TRUE) for implementation-level detail: internal
metric identifiers, the engine settings used for the search, and quantities
recorded on an internal search scale. summary() now prints this detailed
display.complexity and data_control. The top-level predictor_type and
binary_predictor_prevalence arguments have been removed; both are now
data_control fields. See the current examples and function documentation
for details.data_control$correlation = 0.3), whereas 0.5.0 generated independent
predictors. Set correlation = 0 to reproduce the earlier behaviour.pmsims introduces a simulation-based framework for minimum sample size
estimation in prediction model development.simulate_binary(), simulate_continuous(), and
simulate_survival().simulate_custom() interface is available for user-defined
data generators, model-fitting functions, and performance metrics.0.5.0.Any scripts or data that you put into this service are public.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.