bbo-benchmarking: Customized BBo-Benchmarking
In andreas-he/cmaesbenchmarking: Benchmarking of cmaes Stopping Conditions

Description Usage Arguments Details Value Examples

provides an interface for benchmarking optimization algorithms over all 24 noiseless bbob functions. Use either bbob_custom or bbob_custom_parallel for parallel execution.

bbob_custom(optimizer, algorithm_id, data_directory, dimensions = c(2, 3, 5,
  10, 20, 40), instances = c(1:5, 41:50), function_ids = NULL,
  maxit = NULL, stopFitness = NULL, maxFE = NULL, debug.logging = FALSE,
  max_restarts = 0, restart_multiplier = 1,
  restart_triggers = character(0), OCD = FALSE, varLimit = NULL,
  nPreGen = NULL, maxGen = NULL, fitnessValue = FALSE,
  dispersion = FALSE, evolutionPath = FALSE)

bbob_custom_parallel(optimizer, algorithm_id, data_directory,
  dimensions = c(2, 3, 5, 10, 20, 40), instances = c(1:5, 41:50),
  function_ids = NULL, maxit = NULL, stopFitness = NULL, maxFE = NULL,
  debug.logging = FALSE, max_restarts = 0, restart_multiplier = 1,
  restart_triggers = character(0), OCD = FALSE, varLimit = NULL,
  nPreGen = NULL, maxGen = NULL, fitnessValue = FALSE,
  dispersion = FALSE, evolutionPath = FALSE)

`optimizer`	The first argument passed to customized_bbob is an optimization wrapper, i.e. the particular optimizer under test. The optimizer has to adhere to the following function signature: `function(dimension, instance, function_id, maxit, maxFE, stopFitness, path, debug.logging, max_restarts, restart_multiplier, restart_triggers, OCD, varLimit, nPreGen, maxGen, fitnessValue, dispersion, evolutionPath)`. However, the user does not need to specify every argument in an optimizer but it must be able to deal with functions of the type `smoof_function`, that gets `dimension, instance, function_id` as input parameters.
`data_directory`	The `data_directory` specifies the location of the output log files. See detail section for more information.
`dimensions`	The `dimensions` parameter determines the dimensions to be passed to the optimizer under test. Possible values are between 2 and 40. Default is `c(2, 3, 5, 10, 20, 40)`.
`instances`	Every `smoof_function` can be instantiated in different ways. Could be either a single intergerish value greater of equals 1 or a vector with such values. Default is `c(1:15)`
`function_ids`	The `function_ids` are the unique identifier of the 24 noiseless bbob function. Default is `c(1:24)`.
`maxit`	If `maxFE` is not passed, the number of iterations, `maxit`, serves as an upper execution limit for optimizing one function. Default is `NULL`.
`stopFitness`	If `stopFitness` is specified, the optimization is terminated if the gap between the current function value and the global optimum of the function is below the value of `stopFitness`. Default is `NULL`.
`maxFE`	`maxFE` is another upper execution limit for a function optimization. The limit `maxFE` is favored over maxit/stopfitness, if both are not `NULL`. Default is `NULL`.
`debug.logging`	`debug.logging` ... Default is `FALSE`.
`OCD`	`OCD` indicates if Online Convergence Detection (OCD) should be used as a stopping condition of the optimizer under test. OCD can only be activated if the optimizer has implemented the functionality necessary for OCD. See `stopOnOCD` for further necessary parameters. Default is `FALSE`.
`varLimit`	`varLimit`: OCD Parameter (See `stopOnOCD`)
`nPreGen`	`nPreGen`: OCD Parameter (See `stopOnOCD`)
`maxGen`	`maxGen`: OCD Parameter (See `stopOnOCD`)
`fitnessValue`	`fitnessValue`: OCD Parameter (See `stopOnOCD`)
`dispersion`	`disperion`: OCD Parameter (See `stopOnOCD`)
`evolutionPath`	`evolutionPath`: OCD Parameter (See `stopOnOCD`)
`algorithm.id`	The `algorithm.id` is a short descriptive name for the optimization algorithm under test. The `algorithm.id` will be part of the caption of the output log files. See detail section.
`max.restarts`	If `max.restarts` the optimizer restarts the optimization if one of the passed `restart_triggers` has fired. In order to use this feature, the optimizer under test must be able to restart the optimization. Default is `FALSE`.
`restart.triggers`	If `restart.triggers` are passed, the optimizer restarts the optimization if one of those triggers has fired. In order to use this feature, the optimizer under test must be able to restart the optimization. Default is `FALSE`.

bbob_custom provides a benchmarking function as a highlevel interface for running experiments for an optimizer over all 24 noiseless functions of the bbob testset. bbob_custom_parallel does the same in an parallel manner in order to save execution time. The input parameters need to be specified as defined above. Once the execution of an experiment has finished, bbob_custom created a folder according to the datapath specified and writes the results in a number of log files to this folder. The folder's name includes a time stamp as well as the specified data directory (e.g. 2015-12-31_CMAES_default_stopping_conditions). The caption of the log files in that folder is a sequence of the defined algorithm.id, function_ids and dimensions (e.g. CMAES_output_1_2.txt).

bbob_custom does not return anything but writes the results of the experiment to log files, to be processed with readOutput

suppressWarnings(bbob_custom_parallel(optimizer = cmaesbenchmarking::optimizerCMAES, algorithm_id = "CMAES_OCD", 
data_directory = "CMAES_OCD_no_restarts", 
dimensions = c(2, 5, 10, 20), instances = 1:15, function_ids = 1:24, maxit = NULL, 
stopFitness = 1e-08, maxFE = 100000, max_restarts = 0, 
OCD = TRUE, varLimit = 0.0001, nPreGen = 100, fitnessValue = TRUE, 
dispersion = FALSE,  evolutionPath = FALSE, restart_multiplier = 2, 
restart_triggers = "OCD"))