error_handling: Error handling for mlrMBO

In mlrMBO: Bayesian Optimization and Model-Based Optimization of Expensive Black-Box Functions

Error handling for mlrMBO

Description

There are multiple types of errors that can occur during one optimization process. mlrMBO tries to handle most of them as smart as possible.

The target function could

• 1The target function returns NA(s) or NaN(s) (plural for the multi-objective case).

• 2The target function stops with an error.

• 3The target function does not return at all (infinite or very long execution time).

• 4The target function crashes the whole R process.

• 5The surrogate machine learning model might crash. Kriging quite often can run into numerical problems.

• 6The proposal mechanism - in multi-point or single point mode - produces a point which is either close to another candidate point in the same iteration or an already visited point in a previous iteration.

• 7The mbo process exits / stops / crashes itself. Maybe because it hit a walltime.

Mechanism I - Objective value imputation Issues 1-4 all have in common that the optimizer does not obtain a useful objective value. 3-4 are problematic, because we completely lose control of the R process. We are currently only able to handle them, if you are parallelizing your optimization via parallelMap and use the BatchJobs mode. In this case, you can specify a walltime (handles 3) and the function evaluation is performed in a separate R process (handles 4). A later path might be to allow function evaluation in a separate process in general, with a capping time. If you really need this now, you can always do this yourself.

Now back to the problem of invalid objective values. By default, the mbo function stops with an error (if it still has control of the process). But in many cases you still want the algorithm to continue. Hence, mbo allows imputation of bad values via the control option impute.y.fun.

Logging: All error messages are logged into the optimization path opt.path if problems occur.

Mechanism II - The mlr's on.learner.error If your surrogate learner crashes you can set on.surrogate.error in makeMBOControl to “quiet” or “warn”. This will set mlr's on.learner.error for the surrogate. It prevents MBO from crashing in total (issue 5), if the surrogate learner produces an error. As a resort a FailureModel will be returned instead of a the surrogate. Subsequently a random point (or multiple ones) are proposed now for the current iteration. And we pray that we can fit the model again in the next iteration. Logging: The entry “model.error” is set in the opt.path.

Mechanism III - Filtering of proposed point which are too close

Issue 6 is solved by filtering points that are to close to other proposed points or points already proposed in preceding iterations. Filtering in this context means replacing the proposed points by a randomly generated new point. The heuristics mechanism is (de)activated via the logical filter.proposed.points.tol parameter of the setMBOControlInfill function, which defaults to TRUE.(closeness of two points is determined via the filter.proposed.points.tol parameter).

Logging: The logical entry “filtered.point” is set in the opt.path indicating whether the corresponding point was filtered.

Mechanism IV - Continue optimization process

The mechanism is a save-state-then-continue-mechanism, that allows you to continue your optimization after your system or the optimization process crashed for some reason (issue 7). The mbo function has the option to save the current state after certain iterations of the main loop on disk via the control option save.on.disk.at of makeMBOControl. Note that this saving mechanism is disabled by default. Here you can specify, after which iteration you want the current state to be saved (option save.on.disk.at). Notice that 0 denotes saving the initial design and iters + 1 denotes saving the final results. With mboContinue you can continue the optimization from the last saved state. This function only requires the path of the saved state.

You will get a warning if you turn on saving in general, but not for the the final result, as this seems a bit stupid. save.file.path defines the path of the RData file where the state is stored. It is overwritten (= extended) in each saving iteration.

mlrMBO documentation built on July 4, 2022, 9:05 a.m.