race: Racing methods for the selection of the best

In race: Racing methods for the selection of the best

Description

Implementation of some racing methods for the empirical selection of the best. If the R package rpvm is installed (and PVM is available, properly configured, and initialized), the evaluation of the candidates are performed in parallel on different hosts.

Usage

race(wrapper.file, maxExp=0,
        stat.test=c("friedman","t.bonferroni","t.holm","t.none"),
           conf.level=0.95, first.test=5, interactive=TRUE,
              log.file="", no.slaves=0,...)

Arguments

wrapper.file	The name of a file containing the definition of the functions to be provided by the user: i.e. race.wrapper and race.info. The file wrapper.file might also define the functions race.init and race.describe.
maxExp	Maximum number of experiments (i.e. evaluations of the function race.wrapper) that are allowed before selecting the best candidate. If maxExp=0, no limit is imposed... very unrealistic in practice.
stat.test	Statistical test to be used for discarding inferior candidates.
conf.level	The confidence level to be used for the statistical test.
first.test	The first test for discarding inferior candidates is performed only when all candidates have been evaluated on a minimum number of tasks equal to first.test.
interactive	If TRUE, print a progress report on the standard output.
log.file	File for saving periodically the state of the race.
no.slaves	When running under PVM, no.slaves specify the number of slaves to be spawned. If no.slave=0 PVM is not used and all experiments are performed on the local host.
...	All extra parameters are passed to the function race.init defined by the user in the file wrapper.file.

Details

This package implemets some racing procedures for selecting from a set of candidate the one that is able to yield the best performance on a given set of tasks. The time available for selecting the best candidate is limited and, therefore, a brute-force approach is unfeasible. The algorithm implemented in this package sequentially evaluates the set of candidatas on the available tasks while discards bad candidates as soon as statistically sufficient evidence is gathered against them. The elimination of inferior candidates, speeds up the procedure and allows a more reliable evaluation of the promising ones.

Value

The output of race is a list containing the following components:

precis	A string describing the race for documentation purposes.
results	A matrix containing in position [i,j] the result obtained by candidate j on task i.
no.candidates	Number of candidates at the beginning of the race.
no.tasks	Number of tasks on which the selection was based.
no.subtasks	Number of subtasks composing each tasks. Default=1
no.experiments	Number of times that the function race.wrapper had to be call in order to select the best.
no.alive	Number of candidates that completed the race, that is, number of candidates that had not been discarded at the moment in which the race was stopped.
alive	List of the candidates that completed the race: no sufficient evidence was gathered, give that the test stat.test is adopted, for stating that these candidates are worse than the selected best.
alive.inTime	Number of candidates in the race after each time step.
best	The candidate selected in the race.
mean.best	The average result of the best on the tasks considered.
description.best	An object describing the selected candidate.
timestamp.start	Time stamp of the beginning of the race.
timestamp.end	Time stamp of the end of the race.

Note

Please notice that race is a minimization algorithm: it selects the candidate that obtains the smallest results on the various tasks considered.

Author(s)

Mauro Birattari

References

O. Maron and A.W. Moore (1994) Hoeffding Races: Accelerating Model Selection Search for Classification and Function Approximation. Advances in Neural Information Processing Systems 6, pp. 59–66. Morgan Kaufmann.

A.W. Moore and M.S. Lee (1994) Efficient Algorithms for Minimizing Cross Validation Error. International Conference on Machine Learning, pp. 190–198. Morgan Kaufmann.

O. Maron and A.W. Moore (1997) The Racing Algorithm: Model Selection for Lazy Learners. Artificial Intelligence Review, 11(1–5), pp. 193–225.

M. Birattari, T. Stuetzle, L. Paquete, and K. Varrentrapp (2002) A Racing Algorithm for Configuring Metaheuristics. GECCO 2002: Genetic and Evolutionary Computation Conference, pp. 11–18. Morgan Kaufmann.

M. Birattari (2004) The Problem of Tuning Metaheuristics as Seen from a Machine Learning Perspective. PhD Thesis, Universite' Libre de Bruxelles, Brussels, Belgium.

See Also

race.wrapper, race.info, race.init, race.describe

Examples

# The wrapper and init functions for this example are defined in the
# file examples/example-wrapper.R in the installation directory of the
# package.  Please, have a look at such file before implementing your
# own wrapper.
# This example require the package `nnet'
if (require(nnet)&&require(datasets)){
  example.wrapper<-file.path(system.file(package="race"),
                           "examples","example-wrapper.R")
  # Run the race
  race(example.wrapper)

  # If the package `rpvm' is installed on your system and if PVM is
  # properly installed and configured, you can try the following:
  #race(example.wrapper,no.slaves=6)
}

race documentation built on May 29, 2017, 12:03 p.m.