loocvEstimates: Performance estimation using Leave One Out Cross Validation
In performanceEstimation: An Infra-Structure for Performance Estimation of Predictive Models

Description Usage Arguments Details Value Author(s) References See Also Examples

This function obtains leave one out cross validation estimates of performance metrics for a given predictive task and method to solve it (i.e. a workflow). The function is general in the sense that the workflow function that the user provides as the solution to the task, can implement or call whatever modeling technique the user wants.

The function implements leave one out cross validation estimation. Different settings concering this methodology are available through the argument estTask (check the help page of LOOCV).

Please note that most of the times you will not call this function directly, though there is nothing wrong in doing it, but instead you will use the function performanceEstimation, that allows you to carry out performance estimation of multiple workflows on multiple tasks, using some estimation method like for instance cross validation. Still, when you simply want to have the leave one out cross validation estimate of one workflow on one task, you may use this function directly.

1	loocvEstimates(wf,task,estTask,verbose=FALSE,cluster)

`wf`	an object of the class `Workflow` representing the modeling approach to be evaluated on a certain task.
`task`	an object of the class `PredTask` representing the prediction task to be used in the evaluation.
`estTask`	an object of the class `EstimationTask` indicating the metrics to be estimated and the leave one out cross validation settings to use.
`verbose`	A boolean value controlling the level of output of the function execution, defaulting to `FALSE`
`cluster`	an optional parameter that can either be `TRUE` or a `cluster`. In case of `TRUE` the function will run in parallel and will internally setup the parallel back-end (defaulting to using half of the cores in your local machine). You may also setup outside your parallel back-end (c.f. `makeCluster`) and then pass the resulting `cluster` object to this function using this parameter. In case no value is provided for this parameter the function will run sequentially.

The idea of this function is to carry out a leave one out cross validation experiment with the goal of obtaining reliable estimates of the predictive performance of a certain approach to a predictive task. This approach (denoted here as a workflow) will be evaluated on the given predictive task using some user-selected metrics, and this function will provide leave one out cross validation estimates of the true value of these evaluation metrics. Leave one out cross validation estimates are obtained as the average of N iterations, where N is the size of the given data sample. On each of these iterations one of the cases in the data sample is left out as test set and the worflow is applied to the remaining N-1 cases. The process is repeated for all cases, i.e. N times. This estimation is similar to k-fold cross validation where k equals to N. The resulting estimates are obtained by averaging over the N iteration scores.

Parallel execution of the estimation experiment is only recommended for minimally large data sets otherwise you may actually increase the computation time due to communication costs between the processes.

The result of the function is an object of class EstimationResults.

Luis Torgo ltorgo@dcc.fc.up.pt

Torgo, L. (2014) An Infra-Structure for Performance Estimation and Experimental Comparison of Predictive Models in R. arXiv:1412.0436 [cs.MS] http://arxiv.org/abs/1412.0436

LOOCV, Workflow, standardWF, PredTask, EstimationTask, performanceEstimation, hldEstimates, bootEstimates, cvEstimates, mcEstimates, EstimationResults

## Not run: 

## Estimating the error rate of an SVM on the iris data set using
## leave one out cross validation
library(e1071)
data(iris)

## Now the evaluation
eval.res <- loocvEstimates(
             Workflow(wfID="svmTrial",
                      learner="svm",learner.pars=list(cost=10,gamma=0.1)
                     ),
             PredTask(Species ~ ., iris),
             EstimationTask("err",method=LOOCV()))

## Check a summary of the results
summary(eval.res)


## End(Not run)

Task for estimating  err  using
LOOCV experiment
	 Run with seed =  1234 

*** Summary of a  Loocv  Estimation Experiment ***

Task for estimating  err  using
LOOCV experiment
	 Run with seed =  1234 

* Predictive Task ID ::  iris.Species
	Task Type         :: classification 
	Target Feature    :: Species 
	Formula           :: Species ~ .
	Task Data Source  :: iris 

* Workflow        ID ::  svmTrial 
	Workflow Function ::  standardWF
	     Parameter values:
		 learner  ->  svm 
		 learner.pars  ->  cost=10 gamma=0.1 


* Summary of Score Estimation Results:

              err
avg     0.0400000
std     0.1966157
med     0.0000000
iqr     0.0000000
min     0.0000000
max     1.0000000
invalid 0.0000000