tdmBigLoop: Tuning and unbiased evaluation in a big loop.
In TDMR: Tuned Data Mining in R

Description Usage Arguments Details Value Note Author(s) See Also Examples

For each configuration object .conf in tdm$runList call all tuning algorithms (SPOT, CMA-ES or other) specified in tdm$tuneMethod (via function tdmDispatchTuner). After each tuning process perform a run of tdm$unbiasedFunc (usually unbiasedRun).
Each of these experiments is repeated tdm$nExperim times. Thus we have for each tripel

	`(confName,nExp,theTuner)`

a tuning result. The ranges of the triple elements are:

	`confName in tdm$runList`
	`nExp in 1,...,tdm$nExperim`
	`theTuner in tdm$tuneMethod`

1	tdmBigLoop(envT, dataObj = NULL)

envT

an environment containing on input at least the element tdm (a list with general settings for TDMR, see tdmDefaultsFill), which has at least the elements

tdm$runList: vector of configuration names .conf

dataObj

[NULL] optional object of class TDMdata (the same for all runs in big loop). If it is NULL, it will be constructed here with the help of tdmReadAndSplit. Then it can be different for each configuration object in the big loop.

tdm refers to envT$tdm.

The available tuning algorithms (tuners) are

spotTuner: Call spot.
lhdTuner: Perform a parameter tuning using a Latin hypercube design (LHD) for obtaining best design points. LHD is performed by configuring SPOT in such a way that all the budget is used for the initial design (usually LHD).
cma_jTuner: Perform a parameter tuning by CMA-ES, using the *Java* implementation by Niko Hansen through the interface package rCMA-package.
cmaesTuner: Perform a parameter tuning by CMA-ES, using the *R*-implementation (package cma_es by Olaf Mersmann) (deprecated, use cma_jTuner instead).
bfgsTuner: Perform a parameter tuning by Broyden, Fletcher, Goldfarb and Shanno (BFGS) method. The L-BFGS-B version allowing box constraints is used.

environment envT, containing the results

`res`	data frame with results from last tuning (one line for each call of `tdmStart*`)
`bst`	data frame with the best-so-far results from last tuning (one line collected after each (SPO) step)
`resGrid`	list with data frames `res` from all tuning runs. Use `envT$getRes(envT,confFile,nExp,theTuner)` to retrieve a specific `res`.
`bstGrid`	list with data frames `bst` from all tuning runs. Use `envT$getBst(envT,confFile,nExp,theTuner)` to retrieve a specific `bst`.
`theFinals`	data frame with one line for each triple `(confFile,nExp,tuner)`, each line contains summary information about the tuning run in the form: `confFile tuner nExp [params] NRUN NEVAL RGain.bst RGain.* sdR.` where `[params]` is written depending on `tdm$withParams`. `NRUN` is the number of unbiased evaluation runs. `NEVAL` is the number of function evaluations (model builds) during tuning. `RGain` denotes the relative gain on a certain data set: the actual gain achieved with the model divided by the maximum gain possible for the current cost matrix and the current data set. This is for classification tasks, in the case of regression each `RGain.` is replaced by `RMAE.`, the relative mean absolute error. Each 'sdR.' denotes the standard deviation of the preceeding RGain or RMAE. RGain.bst is the best result during tuning obtained on the training-validation data. RGain.avg is the average result during tuning. The following pairs RGain. sdR.* are the results of one or several unbiased evaluations on the test data where '*' takes as many values as there are elements in `tdm$umode` (the possible values are explained in `unbiasedRun`).
`result`	object of class `TDMclassifier` or `TDMregressor`. This is a list with results from `tdm$mainFunc` as called in the last unbiased evaluation using the best parameters found during tuning. Use `print(envT$result)` to get more info on such an object of class `TDMclassifier`.
`tunerVal`	an object with the return value from the last tuning process. For every tuner, this is the list `spotConfig`, containing the SPOT settings plus the TDMR settings in elements `opts` and `tdm`. Every tuner extends this list by `tunerVal$alg.currentResult` and `tunerVal$alg.currentBest`, see `tdmDispatchTuner`. In addition, each tuning method might add specific elements to the list, see the description of each tuner.

Environment envT contains further elements, but they are only relevant for the internal operation of tdmBigLoop and its subfunctions.

Side effects: A compressed version of envT is saved to file tdm$filenameEnvT (default: <runList[1]>.RData) in directory tdm$path. If tdm$path==NULL use the current directory.
If tdm$U.saveModel==TRUE, then envT$result$lastRes$lastModel (the last trained model) will be saved to tdm$filenameEnvT. The default is tdm$U.saveModel==TRUE. If tdm$U.saveModel==FALSE then smaller .RData files will result.

Example usages of function tdmBigLoop are shown in

	`demo(demo03sonar)`
	`demo(demo03sonar_B)`
	`demo(demo04cpu)`

where the corresponding R-sources are in directory demo.

Wolfgang Konen (wolfgang.konen@th-koeln.de), THK, Patrick Koch

tdmDispatchTuner, unbiasedRun

#*# This demo shows a complete tuned data mining process (level 3 of TDMR) where 
#*# the data mining task is the classification task SONAR (from UCI repository, 
#*# http://archive.ics.uci.edu/ml/datasets/Connectionist+Bench+%28Sonar,+Mines+vs.+Rocks%29).
#*# The data mining process is in main_sonar.r, which calls tdmClassifyLoop and tdmClassify
#*# with Random Forest as the prediction model. 
#*# The three parameter to be tuned are CUTOFF1, CLASSWT2 and XPERC, as specified 
#*# in controlSC() (control_sonar.r). The tuner used here is LHD.  
#*# Tuning runs are rather short, to make the example run quickly. 
#*# Do not expect good numeric results. 
#*# See demo/demo03sonar_B.r for a somewhat longer tuning run, with two tuners SPOT and LHD.

  ## path is the dir with data and main_*.r file:
  path <- paste(find.package("TDMR"), "demo02sonar",sep="/");
  #path <- paste("../../inst", "demo02sonar",sep="/");

  ## control settings for TDMR
  tdm <- list( mainFunc="main_sonar"
             , runList = c("sonar_04.conf")
             , umode="CV"              # { "CV" | "RSUB" | "TST" | "SP_T" }
             , tuneMethod = c("lhd")
             , filenameEnvT="exBigLoop.RData"   # file to save environment envT 
             , nrun=1, nfold=2         # repeats and CV-folds for the unbiased runs
             , nExperim=1
             , optsVerbosity = 0       # the verbosity for the unbiased runs
             );
  source(paste(path,"main_sonar.r",sep="/"));    # main_sonar, readTrnSonar


  #*# This demo is for example and help (more meaningful, a bit higher budget)
  source(paste(path,"control_sonar.r",sep="/")); # controlDM, controlSC

  
  
  ctrlSC <- controlSC();
  ctrlSC$opts <- controlDM();
  
  # construct envT from settings given in tdm & sCList
  envT <- tdmEnvTMakeNew(tdm,sCList=list(ctrlSC));
  dataObj <- tdmReadTaskData(envT,envT$tdm);
  envT <- tdmBigLoop(envT,dataObj=dataObj);     # start the big tuning loop