batchmark: Benchmark Experiments on Batch Systems

In mlr3batchmark: Batch Experiments for 'mlr3'

Benchmark Experiments on Batch Systems

Description

This function provides the functionality to leave the interface of mlr3 for the computation of benchmark experiments and switch over to batchtools for a more fine grained control over the execution.

batchmark() populates a batchtools::ExperimentRegistry with jobs in a mlr3::benchmark() fashion. Each combination of mlr3::Task and mlr3::Resampling defines a batchtools::Problem, each mlr3::Learner is an batchtools::Algorithm.

After the jobs have been submitted and are terminated, results can be collected with reduceResultsBatchmark() which returns a mlr3::BenchmarkResult and thus to return to the interface of mlr3.

Usage

batchmark(
  design,
  store_models = FALSE,
  reg = batchtools::getDefaultRegistry(),
  renv_project = NULL
)

Arguments

design	(data.frame()) Data frame (or data.table::data.table()) with three columns: "task", "learner", and "resampling". Each row defines a resampling by providing a Task, Learner and an instantiated Resampling strategy. The helper function benchmark_grid() can assist in generating an exhaustive design (see examples) and instantiate the Resamplings per Task. Additionally, you can set the additional column 'param_values', see benchmark_grid().
store_models	(logical(1)) Store the fitted model in the resulting object= Set to TRUE if you want to further analyse the models or want to extract information like variable importance.
reg	batchtools::ExperimentRegistry.
renv_project	character(1) Path to a renv project. If not NULL, the renv project is activated in the job environment.

Value

data.table::data.table() with ids of created jobs (invisibly).

Examples

tasks = list(mlr3::tsk("iris"), mlr3::tsk("sonar"))
learners = list(mlr3::lrn("classif.featureless"), mlr3::lrn("classif.rpart"))
resamplings = list(mlr3::rsmp("cv", folds = 3), mlr3::rsmp("holdout"))

design = mlr3::benchmark_grid(
  tasks = tasks,
  learners = learners,
  resamplings = resamplings
)

reg = batchtools::makeExperimentRegistry(NA)
batchmark(design, reg = reg)
batchtools::submitJobs(reg = reg)

reduceResultsBatchmark(reg = reg)

mlr3batchmark documentation built on Sept. 9, 2025, 5:51 p.m.