metricsSummary: Obtains a summary of the individual metric scores obtained by...

In performanceEstimation: An Infra-Structure for Performance Estimation of Predictive Models

Description

Given a ComparisonResults object this function provides a summary statistic (defaulting to the mean) of the individual scores obtained on a each evaluation metric over all repetitions carried out in the estimation process. This is done for all workflows and tasks of the performance estimation experiment. The function can be handy to obtain things like for instance the maximum score obtained by each workflow on a particular metric over all repetitions of the experimental process. It is also usefull (using its defaults) as a way to quickly getting the estimated values for each metric obtained by each alternative workflow and task (see the Examples section).

Usage

metricsSummary(compRes, summary = "mean", ...)

Arguments

compRes	An object of class ComparisonResults with the results of a performance estimation experiment.
summary	A string with the name of the function that you want to use to obtain the summary (defaults to "mean"). This function will be applied to the set of individual scores of each workflow on each task and for all metrics.
...	Further arguments passed to the selected summary function.

Value

The result of this function is a named list with as many components as there are predictive tasks. For each task (component), we get a matrix with as many columns as there are workflows and as many rows as there are evaluation metrics. The values on this matrix are the results of applying the selected summary function to the metric scores on each iteration of the estimation process.

Author(s)

Luis Torgo [e-mail]

References

Torgo, L. (2014) An Infra-Structure for Performance Estimation and Experimental Comparison of Predictive Models in R. arXiv:1412.0436 [cs.MS] http://arxiv.org/abs/1412.0436

See Also

performanceEstimation, topPerformers, topPerformer, rankWorkflows

Examples

## Not run: 
## Estimating several evaluation metrics on different variants of a
## regression tree and of a SVM, on  two data sets, using one repetition
## of  10-fold CV

data(swiss)
data(mtcars)
library(e1071)

## run the experimental comparison
results <- performanceEstimation(
               c(PredTask(Infant.Mortality ~ ., swiss),
                 PredTask(mpg ~ ., mtcars)),
               c(workflowVariants(learner='svm',
                                  learner.pars=list(cost=c(1,5),gamma=c(0.1,0.01))
                                 )
               ),
               EstimationTask(metrics=c("mse","mae"),method=CV(nReps=2,nFolds=5))
                                 )

## Get the minium value of each metric on all iterations of the CV
## process. 
metricsSummary(results,summary="min")

## Get a summary table for each task with the estimated scores for each
## metric by each workflow
metricsSummary(results)

## End(Not run)

performanceEstimation documentation built on May 2, 2019, 6:01 a.m.