AutoTuner: Class for Automatic Tuning
In mlr-org/mlr3tuning: Hyperparameter Optimization for 'mlr3'
AutoTuner R Documentation
Class for Automatic Tuning
Description
The AutoTuner wraps a mlr3::Learner and augments it with an automatic tuning process for a given set of hyperparameters.
The auto_tuner() function creates an AutoTuner object.
Details
The AutoTuner is a mlr3::Learner which wraps another mlr3::Learner and performs the following steps during $train():
The hyperparameters of the wrapped (inner) learner are trained on the training data via resampling.
The tuning can be specified by providing a Tuner, a bbotk::Terminator, a search space as paradox::ParamSet, a mlr3::Resampling and a mlr3::Measure.
The best found hyperparameter configuration is set as hyperparameters for the wrapped (inner) learner stored in at$learner.
Access the tuned hyperparameters via at$tuning_result.
A final model is fit on the complete training data using the now parametrized wrapped learner.
The respective model is available via field at$learner$model.
During $predict() the AutoTuner just calls the predict method of the wrapped (inner) learner.
A set timeout is disabled while fitting the final model.
Validation
The AutoTuner itself does not have the "validation" property.
To enable validation during the tuning, set the $validate field of the tuned learner.
This is also possible via set_validate().
Nested Resampling
Nested resampling is performed by passing an AutoTuner to mlr3::resample() or mlr3::benchmark().
To access the inner resampling results, set store_tuning_instance = TRUE and execute mlr3::resample() or mlr3::benchmark() with store_models = TRUE (see examples).
The mlr3::Resampling passed to the AutoTuner is meant to be the inner resampling, operating on the training set of an arbitrary outer resampling.
For this reason, the inner resampling should be not instantiated.
If an instantiated resampling is passed, the AutoTuner fails when a row id of the inner resampling is not present in the training set of the outer resampling.
Default Measures
If no measure is passed, the default measure is used.
The default measure depends on the task type.
Task Default Measure Package
"classif" "classif.ce" mlr3
"regr" "regr.mse" mlr3
"surv" "surv.cindex" mlr3proba
"dens" "dens.logloss" mlr3proba
"classif_st" "classif.ce" mlr3spatial
"regr_st" "regr.mse" mlr3spatial
"clust" "clust.dunn" mlr3cluster
Resources
There are several sections about hyperparameter optimization in the mlr3book.
Getting started with hyperparameter optimization.
An overview of all tuners can be found on our website.
-
Tune a support vector machine on the Sonar data set.
Learn about tuning spaces.
Estimate the model performance with nested resampling.
Learn about multi-objective optimization.
Simultaneously optimize hyperparameters and use early stopping with XGBoost.
-
Automate the tuning.
The gallery features a collection of case studies and demos about optimization.
Learn more advanced methods with the Practical Tuning Series.
Learn about hotstarting models.
Run the default hyperparameter configuration of learners as a baseline.
Use the Hyperband optimizer with different budget parameters.
The cheatsheet summarizes the most important functions of mlr3tuning.
Super class
mlr3::Learner -> AutoTuner
Public fields
instance_args(list())
All arguments from construction to create the TuningInstanceBatchSingleCrit.
tuner(Tuner)
Optimization algorithm.
Active bindings
archiveArchiveBatchTuning
Archive of the TuningInstanceBatchSingleCrit.
learner(mlr3::Learner)
Trained learner
tuning_instance(TuningInstanceAsyncSingleCrit | TuningInstanceBatchSingleCrit)
Internally created tuning instance with all intermediate results.
tuning_result(data.table::data.table)
Short-cut to result from tuning instance.
predict_type(character(1))
Stores the currently active predict type, e.g. "response".
Must be an element of $predict_types.
A few learners already use the predict type during training.
So there is no guarantee that changing the predict type after tuning and training will have any effect or does not lead to errors.
hash(character(1))
Hash (unique identifier) for this object.
phash(character(1))
Hash (unique identifier) for this partial object, excluding some components which are varied systematically during tuning (parameter values) or feature selection (feature names).
Methods
Public methods
-
-
-
-
-
-
-
-
-
-
-
Inherited methods
Method new()
Creates a new instance of this R6 class.
Usage
AutoTuner$new(
tuner,
learner,
resampling,
measure = NULL,
terminator,
search_space = NULL,
store_tuning_instance = TRUE,
store_benchmark_result = TRUE,
store_models = FALSE,
check_values = FALSE,
callbacks = NULL,
rush = NULL,
id = NULL
)
Arguments
tuner(Tuner)
Optimization algorithm.
learner(mlr3::Learner)
Learner to tune.
resampling(mlr3::Resampling)
Resampling that is used to evaluate the performance of the hyperparameter configurations.
Uninstantiated resamplings are instantiated during construction so that all configurations are evaluated on the same data splits.
Already instantiated resamplings are kept unchanged.
Specialized Tuner change the resampling e.g. to evaluate a hyperparameter configuration on different data splits.
This field, however, always returns the resampling passed in construction.
measure(mlr3::Measure)
Measure to optimize. If NULL, default measure is used.
terminator(bbotk::Terminator)
Stop criterion of the tuning process.
search_space(paradox::ParamSet)
Hyperparameter search space. If NULL (default), the search space is
constructed from the paradox::TuneToken of the learner's parameter set
(learner$param_set).
store_tuning_instance(logical(1))
If TRUE (default), stores the internally created TuningInstanceBatchSingleCrit with all intermediate results in slot $tuning_instance.
store_benchmark_result(logical(1))
If TRUE (default), store resample result of evaluated hyperparameter
configurations in archive as mlr3::BenchmarkResult.
store_models(logical(1))
If TRUE, fitted models are stored in the benchmark result
(archive$benchmark_result). If store_benchmark_result = FALSE, models
are only stored temporarily and not accessible after the tuning. This
combination is needed for measures that require a model.
check_values(logical(1))
If TRUE, hyperparameter values are checked before evaluation and
performance scores after. If FALSE (default), values are unchecked but
computational overhead is reduced.
callbacks(list of mlr3misc::Callback)
List of callbacks.
rush(Rush)
If a rush instance is supplied, the tuning runs without batches.
id(character(1))
Identifier for the new instance.
Method base_learner()
Extracts the base learner from nested learner objects like GraphLearner in mlr3pipelines.
If recursive = 0, the (tuned) learner is returned.
Usage
AutoTuner$base_learner(recursive = Inf)
Arguments
recursive(integer(1))
Depth of recursion for multiple nested objects.
Returns
mlr3::Learner.
Method importance()
The importance scores of the final model.
Usage
AutoTuner$importance()
Returns
Named numeric().
Method selected_features()
The selected features of the final model.
Usage
AutoTuner$selected_features()
Returns
character().
Method oob_error()
The out-of-bag error of the final model.
Usage
AutoTuner$oob_error()
Returns
numeric(1).
Method loglik()
The log-likelihood of the final model.
Usage
AutoTuner$loglik()
Returns
logLik.
Printer.
Method print()
Usage
AutoTuner$print()
Arguments
...(ignored).
Method marshal()
Marshal the learner.
Usage
AutoTuner$marshal(...)
Arguments
...(any)
Additional parameters.
Returns
self
Method unmarshal()
Unmarshal the learner.
Usage
AutoTuner$unmarshal(...)
Arguments
...(any)
Additional parameters.
Returns
self
Method marshaled()
Whether the learner is marshaled.
Usage
AutoTuner$marshaled()
Method clone()
The objects of this class are cloneable with this method.
Usage
AutoTuner$clone(deep = FALSE)
Arguments
deepWhether to make a deep clone.
Examples
# Automatic Tuning
# split to train and external set
task = tsk("penguins")
split = partition(task, ratio = 0.8)
# load learner and set search space
learner = lrn("classif.rpart",
cp = to_tune(1e-04, 1e-1, logscale = TRUE)
)
# create auto tuner
at = auto_tuner(
tuner = tnr("random_search"),
learner = learner,
resampling = rsmp ("holdout"),
measure = msr("classif.ce"),
term_evals = 4)
# tune hyperparameters and fit final model
at$train(task, row_ids = split$train)
# predict with final model
at$predict(task, row_ids = split$test)
# show tuning result
at$tuning_result
# model slot contains trained learner and tuning instance
at$model
# shortcut trained learner
at$learner
# shortcut tuning instance
at$tuning_instance
# Nested Resampling
at = auto_tuner(
tuner = tnr("random_search"),
learner = learner,
resampling = rsmp ("holdout"),
measure = msr("classif.ce"),
term_evals = 4)
resampling_outer = rsmp("cv", folds = 3)
rr = resample(task, at, resampling_outer, store_models = TRUE)
# retrieve inner tuning results.
extract_inner_tuning_results(rr)
# performance scores estimated on the outer resampling
rr$score()
# unbiased performance of the final model trained on the full data set
rr$aggregate()
mlr-org/mlr3tuning documentation built on April 14, 2025, 1 a.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
| AutoTuner | R Documentation |
Class for Automatic Tuning
Description
The AutoTuner wraps a mlr3::Learner and augments it with an automatic tuning process for a given set of hyperparameters.
The auto_tuner() function creates an AutoTuner object.
Details
The AutoTuner is a mlr3::Learner which wraps another mlr3::Learner and performs the following steps during $train():
The hyperparameters of the wrapped (inner) learner are trained on the training data via resampling. The tuning can be specified by providing a Tuner, a bbotk::Terminator, a search space as paradox::ParamSet, a mlr3::Resampling and a mlr3::Measure.
The best found hyperparameter configuration is set as hyperparameters for the wrapped (inner) learner stored in
at$learner. Access the tuned hyperparameters viaat$tuning_result.A final model is fit on the complete training data using the now parametrized wrapped learner. The respective model is available via field
at$learner$model.
During $predict() the AutoTuner just calls the predict method of the wrapped (inner) learner.
A set timeout is disabled while fitting the final model.
Validation
The AutoTuner itself does not have the "validation" property.
To enable validation during the tuning, set the $validate field of the tuned learner.
This is also possible via set_validate().
Nested Resampling
Nested resampling is performed by passing an AutoTuner to mlr3::resample() or mlr3::benchmark().
To access the inner resampling results, set store_tuning_instance = TRUE and execute mlr3::resample() or mlr3::benchmark() with store_models = TRUE (see examples).
The mlr3::Resampling passed to the AutoTuner is meant to be the inner resampling, operating on the training set of an arbitrary outer resampling.
For this reason, the inner resampling should be not instantiated.
If an instantiated resampling is passed, the AutoTuner fails when a row id of the inner resampling is not present in the training set of the outer resampling.
Default Measures
If no measure is passed, the default measure is used. The default measure depends on the task type.
| Task | Default Measure | Package |
"classif" | "classif.ce" | mlr3 |
"regr" | "regr.mse" | mlr3 |
"surv" | "surv.cindex" | mlr3proba |
"dens" | "dens.logloss" | mlr3proba |
"classif_st" | "classif.ce" | mlr3spatial |
"regr_st" | "regr.mse" | mlr3spatial |
"clust" | "clust.dunn" | mlr3cluster |
Resources
There are several sections about hyperparameter optimization in the mlr3book.
Getting started with hyperparameter optimization.
An overview of all tuners can be found on our website.
-
Tune a support vector machine on the Sonar data set.
Learn about tuning spaces.
Estimate the model performance with nested resampling.
Learn about multi-objective optimization.
Simultaneously optimize hyperparameters and use early stopping with XGBoost.
-
Automate the tuning.
The gallery features a collection of case studies and demos about optimization.
Learn more advanced methods with the Practical Tuning Series.
Learn about hotstarting models.
Run the default hyperparameter configuration of learners as a baseline.
Use the Hyperband optimizer with different budget parameters.
The cheatsheet summarizes the most important functions of mlr3tuning.
Super class
mlr3::Learner -> AutoTuner
Public fields
instance_args(
list())
All arguments from construction to create the TuningInstanceBatchSingleCrit.tuner(Tuner)
Optimization algorithm.
Active bindings
archiveArchiveBatchTuning
Archive of the TuningInstanceBatchSingleCrit.learner(mlr3::Learner)
Trained learnertuning_instance(TuningInstanceAsyncSingleCrit | TuningInstanceBatchSingleCrit)
Internally created tuning instance with all intermediate results.tuning_result(data.table::data.table)
Short-cut toresultfrom tuning instance.predict_type(
character(1))
Stores the currently active predict type, e.g."response". Must be an element of$predict_types. A few learners already use the predict type during training. So there is no guarantee that changing the predict type after tuning and training will have any effect or does not lead to errors.hash(
character(1))
Hash (unique identifier) for this object.phash(
character(1))
Hash (unique identifier) for this partial object, excluding some components which are varied systematically during tuning (parameter values) or feature selection (feature names).
Methods
Public methods
Inherited methods
Method new()
Creates a new instance of this R6 class.
Usage
AutoTuner$new( tuner, learner, resampling, measure = NULL, terminator, search_space = NULL, store_tuning_instance = TRUE, store_benchmark_result = TRUE, store_models = FALSE, check_values = FALSE, callbacks = NULL, rush = NULL, id = NULL )
Arguments
tuner(Tuner)
Optimization algorithm.learner(mlr3::Learner)
Learner to tune.resampling(mlr3::Resampling)
Resampling that is used to evaluate the performance of the hyperparameter configurations. Uninstantiated resamplings are instantiated during construction so that all configurations are evaluated on the same data splits. Already instantiated resamplings are kept unchanged. Specialized Tuner change the resampling e.g. to evaluate a hyperparameter configuration on different data splits. This field, however, always returns the resampling passed in construction.measure(mlr3::Measure)
Measure to optimize. IfNULL, default measure is used.terminator(bbotk::Terminator)
Stop criterion of the tuning process.search_space(paradox::ParamSet)
Hyperparameter search space. IfNULL(default), the search space is constructed from the paradox::TuneToken of the learner's parameter set (learner$param_set).store_tuning_instance(
logical(1))
IfTRUE(default), stores the internally created TuningInstanceBatchSingleCrit with all intermediate results in slot$tuning_instance.store_benchmark_result(
logical(1))
IfTRUE(default), store resample result of evaluated hyperparameter configurations in archive as mlr3::BenchmarkResult.store_models(
logical(1))
IfTRUE, fitted models are stored in the benchmark result (archive$benchmark_result). Ifstore_benchmark_result = FALSE, models are only stored temporarily and not accessible after the tuning. This combination is needed for measures that require a model.check_values(
logical(1))
IfTRUE, hyperparameter values are checked before evaluation and performance scores after. IfFALSE(default), values are unchecked but computational overhead is reduced.callbacks(list of mlr3misc::Callback)
List of callbacks.rush(
Rush)
If a rush instance is supplied, the tuning runs without batches.id(
character(1))
Identifier for the new instance.
Method base_learner()
Extracts the base learner from nested learner objects like GraphLearner in mlr3pipelines.
If recursive = 0, the (tuned) learner is returned.
Usage
AutoTuner$base_learner(recursive = Inf)
Arguments
recursive(
integer(1))
Depth of recursion for multiple nested objects.
Returns
mlr3::Learner.
Method importance()
The importance scores of the final model.
Usage
AutoTuner$importance()
Returns
Named numeric().
Method selected_features()
The selected features of the final model.
Usage
AutoTuner$selected_features()
Returns
character().
Method oob_error()
The out-of-bag error of the final model.
Usage
AutoTuner$oob_error()
Returns
numeric(1).
Method loglik()
The log-likelihood of the final model.
Usage
AutoTuner$loglik()
Returns
logLik.
Printer.
Method print()
Usage
AutoTuner$print()
Arguments
...(ignored).
Method marshal()
Marshal the learner.
Usage
AutoTuner$marshal(...)
Arguments
...(any)
Additional parameters.
Returns
self
Method unmarshal()
Unmarshal the learner.
Usage
AutoTuner$unmarshal(...)
Arguments
...(any)
Additional parameters.
Returns
self
Method marshaled()
Whether the learner is marshaled.
Usage
AutoTuner$marshaled()
Method clone()
The objects of this class are cloneable with this method.
Usage
AutoTuner$clone(deep = FALSE)
Arguments
deepWhether to make a deep clone.
Examples
# Automatic Tuning
# split to train and external set
task = tsk("penguins")
split = partition(task, ratio = 0.8)
# load learner and set search space
learner = lrn("classif.rpart",
cp = to_tune(1e-04, 1e-1, logscale = TRUE)
)
# create auto tuner
at = auto_tuner(
tuner = tnr("random_search"),
learner = learner,
resampling = rsmp ("holdout"),
measure = msr("classif.ce"),
term_evals = 4)
# tune hyperparameters and fit final model
at$train(task, row_ids = split$train)
# predict with final model
at$predict(task, row_ids = split$test)
# show tuning result
at$tuning_result
# model slot contains trained learner and tuning instance
at$model
# shortcut trained learner
at$learner
# shortcut tuning instance
at$tuning_instance
# Nested Resampling
at = auto_tuner(
tuner = tnr("random_search"),
learner = learner,
resampling = rsmp ("holdout"),
measure = msr("classif.ce"),
term_evals = 4)
resampling_outer = rsmp("cv", folds = 3)
rr = resample(task, at, resampling_outer, store_models = TRUE)
# retrieve inner tuning results.
extract_inner_tuning_results(rr)
# performance scores estimated on the outer resampling
rr$score()
# unbiased performance of the final model trained on the full data set
rr$aggregate()
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.