int_pctl.tune_results: Bootstrap confidence intervals for performance metrics

In tune: Tidy Tuning Tools

Bootstrap confidence intervals for performance metrics

Description

Using out-of-sample predictions, the bootstrap is used to create percentile confidence intervals.

Usage

## S3 method for class 'tune_results'
int_pctl(
  .data,
  metrics = NULL,
  eval_time = NULL,
  times = 1001,
  parameters = NULL,
  alpha = 0.05,
  allow_par = TRUE,
  event_level = "first",
  ...
)

Arguments

.data	A object with class tune_results where the save_pred = TRUE option was used in the control function.
metrics	A yardstick::metric_set(). By default, it uses the same metrics as the original object.
eval_time	A vector of evaluation times for censored regression models. NULL is appropriate otherwise. If NULL is used with censored models, a evaluation time is selected, and a warning is issued.
times	The number of bootstrap samples.
parameters	An optional tibble of tuning parameter values that can be used to filter the predicted values before processing. This tibble should only have columns for each tuning parameter identifier (e.g. "my_param" if tune("my_param") was used).
alpha	Level of significance.
allow_par	A logical to allow parallel processing (if a parallel backend is registered).
event_level	A single string. Either "first" or "second" to specify which level of truth to consider as the "event".
...	Not currently used.

Details

For each model configuration (if any), this function takes bootstrap samples of the out-of-sample predicted values. For each bootstrap sample, the metrics are computed and these are used to compute confidence intervals. See rsample::int_pctl() and the references therein for more details.

Note that the .estimate column is likely to be different from the results given by collect_metrics() since a different estimator is used. Since random numbers are used in sampling, set the random number seed prior to running this function.

The number of bootstrap samples should be large to have reliable intervals. The defaults reflect the fewest samples that should be used.

The computations for each configuration can be extensive. To increase computational efficiency parallel processing can be used. The future package is used here. To execute the resampling iterations in parallel, specify a plan with future first. The allow_par argument can be used to avoid parallelism.

Also, if a censored regression model used numerous evaluation times, the computations can take a long time unless the times are filtered with the eval_time argument.

Value

A tibble of metrics with additional columns for .lower and .upper.

References

Davison, A., & Hinkley, D. (1997). Bootstrap Methods and their Application. Cambridge: Cambridge University Press. doi:10.1017/CBO9780511802843

See Also

rsample::int_pctl()

Examples

data(Sacramento, package = "modeldata")
library(rsample)
library(parsnip)

set.seed(13)
sac_rs <- vfold_cv(Sacramento)

lm_res <-
  linear_reg() %>%
  fit_resamples(
    log10(price) ~ beds + baths + sqft + type + latitude + longitude,
    resamples = sac_rs,
    control = control_resamples(save_pred = TRUE)
  )

set.seed(31)
int_pctl(lm_res)

tune documentation built on May 29, 2024, 7:32 a.m.