irace.cmdline: Launch 'irace' with command-line options.
In irace: Iterated Racing for Automatic Algorithm Configuration

irace.cmdline

R Documentation

Launch `irace` with command-line options.

Description

Calls irace.main() using command-line options, maybe parsed from the command line used to invoke R.

Usage

irace.cmdline(argv = commandArgs(trailingOnly = TRUE))

Arguments

argv

(character())
The arguments provided on the R command line as a character vector, e.g., c("--scenario", "scenario.txt", "-p", "parameters.txt"). Using the default value (not providing the parameter) is the easiest way to call irace.cmdline.

Details

The function reads the parameters given on the command line used to invoke R, finds the name of the scenario file, initializes the scenario from the file (with the function readScenario) and possibly from parameters passed in the command line. It finally starts irace by calling irace.main.

List of command-line options:

-h,--help                Show this help.
-v,--version             Show irace package version.
-c,--check               Check scenario.
-i,--init                Initialize the working directory with template config
                         files.
   --only-test           Only test the configurations given in the file passed
                         as argument.
-s,--scenario            File that describes the configuration scenario setup
                         and other irace settings. Default: ./scenario.txt.
   --exec-dir            Directory where the programs will be run. Default: ./.
-p,--parameter-file      File that contains the description of the parameters
                         of the target algorithm. Default: ./parameters.txt.
   --forbidden-file      File that contains a list of logical expressions that
                         cannot be TRUE for any evaluated configuration. If
                         empty or NULL, do not use forbidden expressions.
   --configurations-file  File that contains a table of initial configurations.
                         If empty or NULL, all initial configurations are
                         randomly generated.
-l,--log-file            File to save tuning results as an R dataset, either
                         absolute path or relative to execDir. Default:
                         ./irace.Rdata.
   --recovery-file       Previously saved log file to recover the execution of
                         irace, either absolute path or relative to the current
                         directory.  If empty or NULL, recovery is not
                         performed.
   --train-instances-dir  Directory where training instances are located;
                         either absolute path or relative to current directory.
                         If no trainInstancesFiles is provided, all the files
                         in trainInstancesDir will be listed as instances.
                         Default: ./Instances.
   --train-instances-file  File that contains a list of training instances and
                         optionally additional parameters for them. If
                         trainInstancesDir is provided, irace will search for
                         the files in this folder.
   --sample-instances    Randomly sample the training instances or use them in
                         the order given. Default: 1.
   --test-instances-dir  Directory where testing instances are located, either
                         absolute or relative to current directory.
   --test-instances-file  File containing a list of test instances and
                         optionally additional parameters for them.
   --test-num-elites     Number of elite configurations returned by irace that
                         will be tested if test instances are provided.
                         Default: 1.
   --test-iteration-elites  Enable/disable testing the elite configurations
                         found at each iteration. Default: 0.
   --test-type           Statistical test used for elimination. The default
                         value selects t-test if capping is enabled or F-test,
                         otherwise. Valid values are: F-test (Friedman test),
                         t-test (pairwise t-tests with no correction),
                         t-test-bonferroni (t-test with Bonferroni's correction
                         for multiple comparisons), t-test-holm (t-test with
                         Holm's correction for multiple comparisons).
   --first-test          Number of instances evaluated before the first
                         elimination test. It must be a multiple of eachTest.
                         Default: 5.
   --each-test           Number of instances evaluated between elimination
                         tests. Default: 1.
   --target-runner       Executable called for each configuration that executes
                         the target algorithm to be tuned. See the templates
                         and examples provided. Default: ./target-runner.
   --target-runner-launcher  Executable that will be used to launch the target
                         runner, when targetRunner cannot be executed directly
                         (.e.g, a Python script in Windows).
   --target-runner-args  Command-line arguments provided to
                         targetRunnerLauncher. The substrings \{targetRunner\}
                         and \{targetRunnerArgs\} will be replaced by the value
                         of the option targetRunner and by the arguments
                         usually passed when calling targetRunner,
                         respectively. Example: "-m {targetRunner --args
                         {targetRunnerArgs}"}. Default: {targetRunner}
                         {targetRunnerArgs}.
   --target-runner-retries  Number of times to retry a call to targetRunner if
                         the call failed. Default: 0.
   --target-evaluator    Optional script or R function that provides a numeric
                         value for each configuration. See
                         templates/target-evaluator.tmpl
   --deterministic       If the target algorithm is deterministic,
                         configurations will be evaluated only once per
                         instance. Default: 0.
   --max-experiments     Maximum number of runs (invocations of targetRunner)
                         that will be performed. It determines the maximum
                         budget of experiments for the tuning. Default: 0.
   --max-time            Maximum total execution time in seconds for the
                         executions of targetRunner. targetRunner must return
                         two values: cost and time. Default: 0.
   --budget-estimation   Fraction (smaller than 1) of the budget used to
                         estimate the mean computation time of a configuration.
                         Only used when maxTime > 0 Default: 0.02.
   --min-measurable-time  Minimum time unit that is still (significantly)
                         measureable. Default: 0.01.
   --parallel            Number of calls to targetRunner to execute in
                         parallel. Values 0 or 1 mean no parallelization.
                         Default: 0.
   --load-balancing      Enable/disable load-balancing when executing
                         experiments in parallel. Load-balancing makes better
                         use of computing resources, but increases
                         communication overhead. If this overhead is large,
                         disabling load-balancing may be faster. Default: 1.
   --mpi                 Enable/disable MPI. Use Rmpi to execute targetRunner
                         in parallel (parameter parallel is the number of
                         slaves). Default: 0.
   --batchmode           Specify how irace waits for jobs to finish when
                         targetRunner submits jobs to a batch cluster: sge,
                         pbs, torque, slurm or htcondor. targetRunner must
                         submit jobs to the cluster using, for example, qsub.
                         Default: 0.
   --digits              Maximum number of decimal places that are significant
                         for numerical (real) parameters. Default: 4.
-q,--quiet               Reduce the output generated by irace to a minimum.
                         Default: 0.
   --debug-level         Debug level of the output of irace. Set this to 0 to
                         silence all debug messages. Higher values provide more
                         verbose debug messages. Default: 0.
   --seed                Seed of the random number generator (by default,
                         generate a random seed).
   --soft-restart        Enable/disable the soft restart strategy that avoids
                         premature convergence of the probabilistic model.
                         Default: 1.
   --soft-restart-threshold  Soft restart threshold value for numerical
                         parameters. If NA, NULL or "", it is computed as
                         10^-digits.
-e,--elitist             Enable/disable elitist irace. Default: 1.
   --elitist-new-instances  Number of instances added to the execution list
                         before previous instances in elitist irace. Default:
                         1.
   --elitist-limit       In elitist irace, maximum number per race of
                         elimination tests that do not eliminate a
                         configuration. Use 0 for no limit. Default: 2.
   --capping             Enable the use of adaptive capping, a technique
                         designed for minimizing the computation time of
                         configurations. This is only available when elitist is
                         active. Default: 0.
   --capping-type        Measure used to obtain the execution bound from the
                         performance of the elite configurations: median, mean,
                         worst, best. Default: median.
   --bound-type          Method to calculate the mean performance of elite
                         configurations: candidate or instance. Default:
                         candidate.
   --bound-max           Maximum execution bound for targetRunner. It must be
                         specified when capping is enabled. Default: 0.
   --bound-digits        Precision used for calculating the execution time. It
                         must be specified when capping is enabled. Default: 0.
   --bound-par           Penalization constant for timed out executions
                         (executions that reach boundMax execution time).
                         Default: 1.
   --bound-as-timeout    Replace the configuration cost of bounded executions
                         with boundMax. Default: 1.
   --postselection       Percentage of the configuration budget used to perform
                         a postselection race of the best configurations of
                         each iteration after the execution of irace. Default:
                         0.
   --aclib               Enable/disable AClib mode. This option enables
                         compatibility with GenericWrapper4AC as targetRunner
                         script. Default: 0.
   --iterations          Maximum number of iterations. Default: 0.
   --experiments-per-iteration  Number of runs of the target algorithm per
                         iteration. Default: 0.
   --min-survival        Minimum number of configurations needed to continue
                         the execution of each race (iteration). Default: 0.
   --num-configurations  Number of configurations to be sampled and evaluated
                         at each iteration. Default: 0.
   --mu                  Parameter used to define the number of configurations
                         sampled and evaluated at each iteration. Default: 5.
   --confidence          Confidence level for the elimination test. Default:
                         0.95.

Value

(invisible(data.frame))

A data frame with the set of best algorithm configurations found by irace. The data frame has the following columns:

.ID. : Internal id of the candidate configuration.
Parameter names : One column per parameter name in parameters.
.PARENT. : Internal id of the parent candidate configuration.

Additionally, this function saves an R data file containing an object called iraceResults. The path of the file is indicated in scenario$logFile. The iraceResults object is a list with the following structure:

scenario: The scenario R object containing the irace options used for the execution. See defaultScenario for more information.
parameters: The parameters R object containing the description of the target algorithm parameters. See readParameters.
allConfigurations: The target algorithm configurations generated by irace. This object is a data frame, each row is a candidate configuration, the first column (.ID.) indicates the internal identifier of the configuration, the following columns correspond to the parameter values, each column named as the parameter name specified in the parameter object. The final column (.PARENT.) is the identifier of the configuration from which model the actual configuration was sampled.
allElites: A list that contains one element per iteration, each element contains the internal identifier of the elite candidate configurations of the corresponding iteration (identifiers correspond to allConfigurations$.ID.).
iterationElites: A vector containing the best candidate configuration internal identifier of each iteration. The best configuration found corresponds to the last one of this vector.
experiments: A matrix with configurations as columns and instances as rows. Column names correspond to the internal identifier of the configuration (allConfigurations$.ID.).
experimentLog: A matrix with columns iteration, instance, configuration, time. This matrix contains the log of all the experiments that irace performs during its execution. The instance column refers to the index of the scenario$instancesList data frame. Time is saved ONLY when reported by the targetRunner.
softRestart: A logical vector that indicates if a soft restart was performed on each iteration. If FALSE, then no soft restart was performed.
state: A list that contains the state of irace, the recovery is done using the information contained in this object.
testing: A list that contains the testing results. The elements of this list are: experiments a matrix with the testing experiments of the selected configurations in the same format as the explained above and seeds a vector with the seeds used to execute each experiment.