benchmark: Benchmark experiment for multiple learners and tasks.

Description Usage Arguments Value See Also Examples

View source: R/benchmark.R

Description

Complete benchmark experiment to compare different learning algorithms across one or more tasks w.r.t. a given resampling strategy. Experiments are paired, meaning always the same training / test sets are used for the different learners. Furthermore, you can of course pass “enhanced” learners via wrappers, e.g., a learner can be automatically tuned using makeTuneWrapper.

Usage

1
2
benchmark(learners, tasks, resamplings, measures, keep.pred = TRUE,
  models = TRUE, show.info = getMlrOption("show.info"))

Arguments

learners

[(list of) Learner | character]
Learning algorithms which should be compared, can also be a single learner. If you pass strings the learners will be created via makeLearner.

tasks

[(list of) Task]
Tasks that learners should be run on.

resamplings

[(list of) ResampleDesc | ResampleInstance]
Resampling strategy for each tasks. If only one is provided, it will be replicated to match the number of tasks. If missing, a 10-fold cross validation is used.

measures

[(list of) Measure]
Performance measures for all tasks. If missing, the default measure of the first task is used.

keep.pred

[logical(1)]
Keep the prediction data in the pred slot of the result object. If you do many experiments (on larger data sets) these objects might unnecessarily increase object size / mem usage, if you do not really need them. In this case you can set this argument to FALSE. Default is TRUE.

models

[logical(1)]
Should all fitted models be stored in the ResampleResult? Default is TRUE.

show.info

[logical(1)]
Print verbose output on console? Default is set via configureMlr.

Value

[BenchmarkResult].

See Also

Other benchmark: BenchmarkResult, batchmark, convertBMRToRankMatrix, friedmanPostHocTestBMR, friedmanTestBMR, generateCritDifferencesData, getBMRAggrPerformances, getBMRFeatSelResults, getBMRFilteredFeatures, getBMRLearnerIds, getBMRLearnerShortNames, getBMRLearners, getBMRMeasureIds, getBMRMeasures, getBMRModels, getBMRPerformances, getBMRPredictions, getBMRTaskDescs, getBMRTaskIds, getBMRTuneResults, plotBMRBoxplots, plotBMRRanksAsBarChart, plotBMRSummary, plotCritDifferences, reduceBatchmarkResults

Examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
lrns = list(makeLearner("classif.lda"), makeLearner("classif.rpart"))
tasks = list(iris.task, sonar.task)
rdesc = makeResampleDesc("CV", iters = 2L)
meas = list(acc, ber)
bmr = benchmark(lrns, tasks, rdesc, measures = meas)
rmat = convertBMRToRankMatrix(bmr)
print(rmat)
plotBMRSummary(bmr)
plotBMRBoxplots(bmr, ber, style = "violin")
plotBMRRanksAsBarChart(bmr, pos = "stack")
friedmanTestBMR(bmr)
friedmanPostHocTestBMR(bmr, p.value = 0.05)

Example output

Loading required package: ParamHelpers
Task: iris-example, Learner: classif.lda
[Resample] cross-validation iter 1: acc.test.mean=0.973,ber.test.mean=0.0274
[Resample] cross-validation iter 2: acc.test.mean=0.987,ber.test.mean=0.0159
[Resample] Aggr. Result: acc.test.mean=0.98,ber.test.mean=0.0216
Task: Sonar-example, Learner: classif.lda
[Resample] cross-validation iter 1: acc.test.mean=0.702,ber.test.mean=0.31
[Resample] cross-validation iter 2: acc.test.mean=0.721,ber.test.mean=0.28
[Resample] Aggr. Result: acc.test.mean=0.712,ber.test.mean=0.295
Task: iris-example, Learner: classif.rpart
[Resample] cross-validation iter 1: acc.test.mean=0.92,ber.test.mean=0.0821
[Resample] cross-validation iter 2: acc.test.mean=0.96,ber.test.mean=0.0389
[Resample] Aggr. Result: acc.test.mean=0.94,ber.test.mean=0.0605
Task: Sonar-example, Learner: classif.rpart
[Resample] cross-validation iter 1: acc.test.mean=0.692,ber.test.mean=0.336
[Resample] cross-validation iter 2: acc.test.mean=0.692,ber.test.mean=0.307
[Resample] Aggr. Result: acc.test.mean=0.692,ber.test.mean=0.322
              Sonar-example iris-example
classif.lda               1            1
classif.rpart             2            2
Warning messages:
1: In max(data$density) : no non-missing arguments to max; returning -Inf
2: In max(data$density) : no non-missing arguments to max; returning -Inf

	Friedman rank sum test

data:  acc.test.mean and learner.id and task.id
Friedman chi-squared = 2, df = 1, p-value = 0.1573

Loading required package: PMCMR

	Friedman rank sum test

data:  acc.test.mean and learner.id and task.id
Friedman chi-squared = 2, df = 1, p-value = 0.1573

Warning message:
In friedmanPostHocTestBMR(bmr, p.value = 0.05) :
  Cannot reject null hypothesis of overall Friedman test,
             returning overall Friedman test.

mlr documentation built on May 29, 2017, 6:01 p.m.