rankWorkflows: Provide a ranking of workflows involved in an estimation...
In performanceEstimation: An Infra-Structure for Performance Estimation of Predictive Models

Description Usage Arguments Value Author(s) References See Also Examples

Given a ComparisonResults object resulting from a performance estimation experiment, this function provides a ranking (by default the top 5) of the best workflows involved in the comparison. The rankings are provided by task and for each evaluation metric.

1 2	rankWorkflows(compRes,top=min(5,length(workflowNames(compRes))), maxs=rep(FALSE,dim(compRes[[1]][[1]]@iterationsScores)[2]),stat="avg")

`compRes`	An object of class `ComparisonResults` with the results of the performance estimation experiment.
`top`	The number of workflows to include in the rankings (defaulting to 5 or the number of workflows in the experiment if less than 5)
`maxs`	A vector of booleans with as many elements are there are statistics measured in the experimental comparison. A `TRUE` value means the respective metric is to be maximized, while a `FALSE` means minimization. Defaults to all `FALSE` values.
`stat`	The statistic to be used to obtain the ranks. The options are the statistics produced by the function `summary` applied to objects of class `ComparisonResults`, i.e. "avg", "std", "med", "iqr", "min", "max" or "invalid" (defaults to "avg").

The function returns a named list with as many components as there are predictive tasks in the experiment. For each task you will get another named list, with as many elements as there evaluation metrics. For each of these components you have a data frame with N lines, where N is the size of the requested rank. Each line includes the name of the workflow in the respective rank position and the estimated score it got on that particular task / evaluation metric.

Luis Torgo ltorgo@dcc.fc.up.pt

Torgo, L. (2014) An Infra-Structure for Performance Estimation and Experimental Comparison of Predictive Models in R. arXiv:1412.0436 [cs.MS] http://arxiv.org/abs/1412.0436

performanceEstimation, topPerformers, topPerformer, metricsSummary

## Not run: 
## Estimating several evaluation metrics on different variants of a
## regression tree and of a SVM, on  two data sets, using one repetition
## of  10-fold CV

data(swiss)
data(mtcars)
library(e1071)

## run the experimental comparison
results <- performanceEstimation(
               c(PredTask(Infant.Mortality ~ ., swiss),
                 PredTask(mpg ~ ., mtcars)),
               c(workflowVariants(learner='svm',
                                  learner.pars=list(cost=c(1,5),gamma=c(0.1,0.01))
                                 )
               ),
               EstimationTask(metrics=c("mse","mae"),method=CV(nReps=2,nFolds=5))
                                 )
## get a ranking of the top workflows for each task and evaluation metric
rankWorkflows(results)
## get a ranking of the top workflows for each task and evaluation
## metric by the median score on all iterations instead of the mean score
rankWorkflows(results, stat="med")

## End(Not run)

##### PERFORMANCE ESTIMATION USING  CROSS VALIDATION  #####

** PREDICTIVE TASK :: swiss.Infant.Mortality

++ MODEL/WORKFLOW :: svm.v1 
Task for estimating  mse,mae  using
 2 x 5 - Fold Cross Validation
	 Run with seed =  1234 
Iteration :**********


++ MODEL/WORKFLOW :: svm.v2 
Task for estimating  mse,mae  using
 2 x 5 - Fold Cross Validation
	 Run with seed =  1234 
Iteration :**********


++ MODEL/WORKFLOW :: svm.v3 
Task for estimating  mse,mae  using
 2 x 5 - Fold Cross Validation
	 Run with seed =  1234 
Iteration :**********


++ MODEL/WORKFLOW :: svm.v4 
Task for estimating  mse,mae  using
 2 x 5 - Fold Cross Validation
	 Run with seed =  1234 
Iteration :**********


** PREDICTIVE TASK :: mtcars.mpg

++ MODEL/WORKFLOW :: svm.v1 
Task for estimating  mse,mae  using
 2 x 5 - Fold Cross Validation
	 Run with seed =  1234 
Iteration :**********


++ MODEL/WORKFLOW :: svm.v2 
Task for estimating  mse,mae  using
 2 x 5 - Fold Cross Validation
	 Run with seed =  1234 
Iteration :**********


++ MODEL/WORKFLOW :: svm.v3 
Task for estimating  mse,mae  using
 2 x 5 - Fold Cross Validation
	 Run with seed =  1234 
Iteration :**********


++ MODEL/WORKFLOW :: svm.v4 
Task for estimating  mse,mae  using
 2 x 5 - Fold Cross Validation
	 Run with seed =  1234 
Iteration :**********
$swiss.Infant.Mortality
$swiss.Infant.Mortality$mse
  Workflow Estimate
1   svm.v2 7.236643
2   svm.v4 7.418691
3   svm.v1 7.420137
4   svm.v3 7.892514

$swiss.Infant.Mortality$mae
  Workflow Estimate
1   svm.v2 2.062452
2   svm.v4 2.085604
3   svm.v3 2.102002
4   svm.v1 2.135031


$mtcars.mpg
$mtcars.mpg$mse
  Workflow Estimate
1   svm.v4 11.74371
2   svm.v3 12.44558
3   svm.v2 16.37139
4   svm.v1 17.07277

$mtcars.mpg$mae
  Workflow Estimate
1   svm.v3 2.667208
2   svm.v4 2.722154
3   svm.v2 3.188510
4   svm.v1 3.212202


$swiss.Infant.Mortality
$swiss.Infant.Mortality$mse
  Workflow Estimate
1   svm.v4 5.702219
2   svm.v1 5.750239
3   svm.v2 6.005752
4   svm.v3 6.163613

$swiss.Infant.Mortality$mae
  Workflow Estimate
1   svm.v1 1.997416
2   svm.v4 2.013057
3   svm.v3 2.017734
4   svm.v2 2.115445


$mtcars.mpg
$mtcars.mpg$mse
  Workflow  Estimate
1   svm.v4  7.861549
2   svm.v3 10.153903
3   svm.v1 11.785147
4   svm.v2 12.554266

$mtcars.mpg$mae
  Workflow Estimate
1   svm.v4 2.289458
2   svm.v1 2.689282
3   svm.v3 2.730878
4   svm.v2 2.904331