irace_cmdline: Launch 'irace' with command-line options.
In irace: Iterated Racing for Automatic Algorithm Configuration

irace_cmdline

R Documentation

Launch `irace` with command-line options.

Description

Calls irace_main() using command-line options, maybe parsed from the command line used to invoke R.

Usage

irace_cmdline(argv = commandArgs(trailingOnly = TRUE))

irace.cmdline(argv = commandArgs(trailingOnly = TRUE))

Arguments

argv

character()
The arguments provided on the R command line as a character vector, e.g., c("--scenario", "scenario.txt", "-p", "parameters.txt"). Using the default value (not providing the parameter) is the easiest way to call irace_cmdline().

Details

The function reads the parameters given on the command line used to invoke R, finds the name of the scenario file, initializes the scenario from the file (with the function readScenario()) and possibly from parameters passed in the command line. It finally starts irace by calling irace_main().

List of command-line options:

-h,--help                Show this help.
-v,--version             Show irace package version.
-c,--check               Check scenario.
-i,--init                Initialize the working directory with template config
                         files.
   --only-test           Only test the configurations given in the file passed
                         as argument.
-s,--scenario            File that describes the configuration scenario setup
                         and other irace settings. Default: ./scenario.txt.
   --exec-dir            Directory where the programs will be run. Default: ./.
-p,--parameter-file      File that contains the description of the parameters
                         of the target algorithm. Default: ./parameters.txt.
   --configurations-file  File that contains a table of initial configurations.
                         If empty or `NULL`, all initial configurations are
                         randomly generated.
-l,--log-file            File to save tuning results as an R dataset, either
                         absolute path or relative to execDir. Default:
                         ./irace.Rdata.
   --recovery-file       Previously saved log file to recover the execution of
                         `irace`, either absolute path or relative to the
                         current directory.  If empty or `NULL`, recovery is
                         not performed.
   --train-instances-dir  Directory where training instances are located;
                         either absolute path or relative to current directory.
                         If no `trainInstancesFiles` is provided, all the files
                         in `trainInstancesDir` will be listed as instances.
   --train-instances-file  File that contains a list of training instances and
                         optionally additional parameters for them. If
                         `trainInstancesDir` is provided, `irace` will search
                         for the files in this folder.
   --sample-instances    Randomly sample the training instances or use them in
                         the order given. Default: 1.
   --test-instances-dir  Directory where testing instances are located, either
                         absolute or relative to current directory.
   --test-instances-file  File containing a list of test instances and
                         optionally additional parameters for them.
   --test-num-elites     Number of elite configurations returned by irace that
                         will be tested if test instances are provided.
                         Default: 1.
   --test-iteration-elites  Enable/disable testing the elite configurations
                         found at each iteration. Default: 0.
   --test-type           Statistical test used for elimination. The default
                         value selects `t-test` if `capping` is enabled or
                         `F-test`, otherwise. Valid values are: F-test
                         (Friedman test), t-test (pairwise t-tests with no
                         correction), t-test-bonferroni (t-test with
                         Bonferroni's correction for multiple comparisons),
                         t-test-holm (t-test with Holm's correction for
                         multiple comparisons).
   --first-test          Number of instances evaluated before the first
                         elimination test. It must be a multiple of `eachTest`.
                         Default: 5.
   --block-size          Number of training instances, that make up a 'block'
                         in `trainInstancesFile`. Elimination of configurations
                         will only be performed after evaluating a complete
                         block and never in the middle of a block. Each block
                         typically contains one instance from each instance
                         class (type or family) and the block size is the
                         number of classes. The value of `blockSize` will
                         multiply `firstTest`, `eachTest` and
                         `elitistNewInstances`. Default: 1.
   --each-test           Number of instances evaluated between elimination
                         tests. Default: 1.
   --target-runner       Executable called for each configuration that executes
                         the target algorithm to be tuned. See the templates
                         and examples provided. Default: ./target-runner.
   --target-runner-launcher  Executable that will be used to launch the target
                         runner, when `targetRunner` cannot be executed
                         directly (e.g., a Python script in Windows).
   --target-cmdline      Command-line arguments provided to `targetRunner` (or
                         `targetRunnerLauncher` if defined). The substrings
                         `\{configurationID\}`, `\{instanceID\}`, `\{seed\}`,
                         `\{instance\}`, and `\{bound\}` will be replaced by
                         their corresponding values. The substring
                         `\{targetRunnerArgs\}` will be replaced by the
                         concatenation of the switch and value of all active
                         parameters of the particular configuration being
                         evaluated.  The substring `\{targetRunner\}`, if
                         present, will be replaced by the value of
                         `targetRunner` (useful when using
                         `targetRunnerLauncher`). Default: {configurationID}
                         {instanceID} {seed} {instance} {bound}
                         {targetRunnerArgs}.
   --target-runner-retries  Number of times to retry a call to `targetRunner`
                         if the call failed. Default: 0.
   --target-runner-timeout  Timeout in seconds of any `targetRunner` call (only
                         applies to `target-runner` executables not to R
                         functions), ignored if 0. Default: 0.
   --target-evaluator    Optional script or R function that provides a numeric
                         value for each configuration. See
                         templates/target-evaluator.tmpl
   --deterministic       If the target algorithm is deterministic,
                         configurations will be evaluated only once per
                         instance. Default: 0.
   --max-experiments     Maximum number of runs (invocations of `targetRunner`)
                         that will be performed. It determines the maximum
                         budget of experiments for the tuning. Default: 0.
   --min-experiments     Minimum number of runs (invocations of `targetRunner`)
                         that will be performed. It determines the minimum
                         budget of experiments for the tuning. The actual
                         budget depends on the number of parameters and
                         `minSurvival`.
   --max-time            Maximum total execution time for the executions of
                         `targetRunner`. `targetRunner` must return two values:
                         cost and time. This value and the one returned by
                         `targetRunner` must use the same units (seconds,
                         minutes, iterations, evaluations, ...). Default: 0.
   --budget-estimation   Fraction (smaller than 1) of the budget used to
                         estimate the mean computation time of a configuration.
                         Only used when `maxTime` > 0 Default: 0.05.
   --min-measurable-time  Minimum time unit that is still (significantly)
                         measureable. Default: 0.01.
   --parallel            Number of calls to `targetRunner` to execute in
                         parallel. Values `0` or `1` mean no parallelization.
                         Default: 0.
   --load-balancing      Enable/disable load-balancing when executing
                         experiments in parallel. Load-balancing makes better
                         use of computing resources, but increases
                         communication overhead. If this overhead is large,
                         disabling load-balancing may be faster. Default: 1.
   --mpi                 Enable/disable MPI. Use `Rmpi` to execute
                         `targetRunner` in parallel (parameter `parallel` is
                         the number of slaves). Default: 0.
   --batchmode           Specify how irace waits for jobs to finish when
                         `targetRunner` submits jobs to a batch cluster: sge,
                         pbs, torque, slurm or htcondor. `targetRunner` must
                         submit jobs to the cluster using, for example, `qsub`.
                         Default: 0.
-q,--quiet               Reduce the output generated by irace to a minimum.
                         Default: 0.
   --debug-level         Debug level of the output of `irace`. Set this to 0 to
                         silence all debug messages. Higher values provide more
                         verbose debug messages. Default: 0.
   --seed                Seed of the random number generator (by default,
                         generate a random seed).
   --soft-restart        Enable/disable the soft restart strategy that avoids
                         premature convergence of the probabilistic model.
                         Default: 1.
   --soft-restart-threshold  Soft restart threshold value for numerical
                         parameters. Default: 1e-04.
-e,--elitist             Enable/disable elitist irace. Default: 1.
   --elitist-new-instances  Number of instances added to the execution list
                         before previous instances in elitist irace. Default:
                         1.
   --elitist-limit       In elitist irace, maximum number per race of
                         elimination tests that do not eliminate a
                         configuration. Use 0 for no limit. Default: 2.
   --capping             Enable the use of adaptive capping, a technique
                         designed for minimizing the computation time of
                         configurations. Capping is enabled by default if
                         `elitist` is active, `maxTime > 0` and `boundMax > 0`.
   --capping-after-first-test  If set to 1, elimination due to capping only
                         happens after `firstTest` instances are seen. Default:
                         0.
   --capping-type        Measure used to obtain the execution bound from the
                         performance of the elite configurations: median, mean,
                         worst, best. Default: median.
   --bound-type          Method to calculate the mean performance of elite
                         configurations: candidate or instance. Default:
                         candidate.
   --bound-max           Maximum execution bound for `targetRunner`. It must be
                         specified when capping is enabled. Default: 0.
   --bound-digits        Precision used for calculating the execution time. It
                         must be specified when capping is enabled. Default: 0.
   --bound-par           Penalization constant for timed out executions
                         (executions that reach `boundMax` execution time).
                         Default: 1.
   --bound-as-timeout    Replace the configuration cost of bounded executions
                         with `boundMax`. Default: 1.
   --postselection       Perform a postselection race after the execution of
                         irace to consume all remaining budget. Value 0
                         disables the postselection race. Default: 1.
   --aclib               Enable/disable AClib mode. This option enables
                         compatibility with GenericWrapper4AC as targetRunner
                         script. Default: 0.
   --iterations          Maximum number of iterations. Default: 0.
   --experiments-per-iteration  Number of runs of the target algorithm per
                         iteration. Default: 0.
   --min-survival        Minimum number of configurations needed to continue
                         the execution of each race (iteration). Default: 0.
   --num-configurations  Number of configurations to be sampled and evaluated
                         at each iteration. Default: 0.
   --mu                  Parameter used to define the number of configurations
                         sampled and evaluated at each iteration. Default: 5.
   --confidence          Confidence level for the elimination test. Default:
                         0.95.

Value

(invisible(data.frame))

A data frame with the set of best algorithm configurations found by irace. The data frame has the following columns:

.ID. : Internal id of the candidate configuration.
⁠Parameter names⁠ : One column per parameter name in parameters.
.PARENT. : Internal id of the parent candidate configuration.

Additionally, this function saves an R data file containing an object called iraceResults. The path of the file is indicated in scenario$logFile. The iraceResults object is a list with the following structure:

scenario: The scenario R object containing the irace options used for the execution. See defaultScenario for more information. The element scenario$parameters contains the parameters R object that describes the target algorithm parameters. See readParameters.
allConfigurations: The target algorithm configurations generated by irace. This object is a data frame, each row is a candidate configuration, the first column (.ID.) indicates the internal identifier of the configuration, the following columns correspond to the parameter values, each column named as the parameter name specified in the parameter object. The final column (.PARENT.) is the identifier of the configuration from which model the actual configuration was sampled.
allElites: A list that contains one element per iteration, each element contains the internal identifier of the elite candidate configurations of the corresponding iteration (identifiers correspond to allConfigurations$.ID.).
iterationElites: A vector containing the best candidate configuration internal identifier of each iteration. The best configuration found corresponds to the last one of this vector.
experiments: A matrix with configurations as columns and instances as rows. Column names correspond to the internal identifier of the configuration (allConfigurations$.ID.).
experimen_log: A data.table with columns iteration, instance, configuration, time. This matrix contains the log of all the experiments that irace performs during its execution. The instance column refers to the index of the race_state$instances_log data frame. Time is saved ONLY when reported by the targetRunner.
softRestart: A logical vector that indicates if a soft restart was performed on each iteration. If FALSE, then no soft restart was performed.
state: An environment that contains the state of irace, the recovery is done using the information contained in this object.
testing: A list that contains the testing results. The elements of this list are: experiments a matrix with the testing experiments of the selected configurations in the same format as the explained above and seeds a vector with the seeds used to execute each experiment.