benchmark: Benchmark experiment for multiple learners and tasks.

Description Usage Arguments Value See Also Examples

View source: R/benchmark.R

Description

Complete benchmark experiment to compare different learning algorithms across one or more tasks w.r.t. a given resampling strategy. Experiments are paired, meaning always the same training / test sets are used for the different learners. Furthermore, you can of course pass “enhanced” learners via wrappers, e.g., a learner can be automatically tuned using makeTuneWrapper.

Usage

1
2
benchmark(learners, tasks, resamplings, measures, keep.pred = TRUE,
  models = TRUE, show.info = getMlrOption("show.info"))

Arguments

learners

(list of Learner | character)
Learning algorithms which should be compared, can also be a single learner. If you pass strings the learners will be created via makeLearner.

tasks

list of Task
Tasks that learners should be run on.

resamplings

(list of ResampleDesc | ResampleInstance)
Resampling strategy for each tasks. If only one is provided, it will be replicated to match the number of tasks. If missing, a 10-fold cross validation is used.

measures

(list of Measure)
Performance measures for all tasks. If missing, the default measure of the first task is used.

keep.pred

(logical(1))
Keep the prediction data in the pred slot of the result object. If you do many experiments (on larger data sets) these objects might unnecessarily increase object size / mem usage, if you do not really need them. In this case you can set this argument to FALSE. Default is TRUE.

models

(logical(1))
Should all fitted models be stored in the ResampleResult? Default is TRUE.

show.info

(logical(1))
Print verbose output on console? Default is set via configureMlr.

Value

BenchmarkResult.

See Also

Other benchmark: BenchmarkResult, batchmark, convertBMRToRankMatrix, friedmanPostHocTestBMR, friedmanTestBMR, generateCritDifferencesData, getBMRAggrPerformances, getBMRFeatSelResults, getBMRFilteredFeatures, getBMRLearnerIds, getBMRLearnerShortNames, getBMRLearners, getBMRMeasureIds, getBMRMeasures, getBMRModels, getBMRPerformances, getBMRPredictions, getBMRTaskDescs, getBMRTaskIds, getBMRTuneResults, plotBMRBoxplots, plotBMRRanksAsBarChart, plotBMRSummary, plotCritDifferences, reduceBatchmarkResults

Examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
lrns = list(makeLearner("classif.lda"), makeLearner("classif.rpart"))
tasks = list(iris.task, sonar.task)
rdesc = makeResampleDesc("CV", iters = 2L)
meas = list(acc, ber)
bmr = benchmark(lrns, tasks, rdesc, measures = meas)
rmat = convertBMRToRankMatrix(bmr)
print(rmat)
plotBMRSummary(bmr)
plotBMRBoxplots(bmr, ber, style = "violin")
plotBMRRanksAsBarChart(bmr, pos = "stack")
friedmanTestBMR(bmr)
friedmanPostHocTestBMR(bmr, p.value = 0.05)

Example output

Loading required package: ParamHelpers
Task: iris-example, Learner: classif.lda
[Resample] cross-validation iter 1: acc.test.mean=0.973,ber.test.mean=0.0274
[Resample] cross-validation iter 2: acc.test.mean=0.987,ber.test.mean=0.0159
[Resample] Aggr. Result: acc.test.mean=0.98,ber.test.mean=0.0216
Task: Sonar-example, Learner: classif.lda
[Resample] cross-validation iter 1: acc.test.mean=0.702,ber.test.mean=0.31
[Resample] cross-validation iter 2: acc.test.mean=0.721,ber.test.mean=0.28
[Resample] Aggr. Result: acc.test.mean=0.712,ber.test.mean=0.295
Task: iris-example, Learner: classif.rpart
[Resample] cross-validation iter 1: acc.test.mean=0.92,ber.test.mean=0.0821
[Resample] cross-validation iter 2: acc.test.mean=0.96,ber.test.mean=0.0389
[Resample] Aggr. Result: acc.test.mean=0.94,ber.test.mean=0.0605
Task: Sonar-example, Learner: classif.rpart
[Resample] cross-validation iter 1: acc.test.mean=0.692,ber.test.mean=0.336
[Resample] cross-validation iter 2: acc.test.mean=0.692,ber.test.mean=0.307
[Resample] Aggr. Result: acc.test.mean=0.692,ber.test.mean=0.322
              Sonar-example iris-example
classif.lda               1            1
classif.rpart             2            2
Warning messages:
1: In max(data$density) : no non-missing arguments to max; returning -Inf
2: In max(data$density) : no non-missing arguments to max; returning -Inf

	Friedman rank sum test

data:  acc.test.mean and learner.id and task.id
Friedman chi-squared = 2, df = 1, p-value = 0.1573

Loading required package: PMCMR

	Friedman rank sum test

data:  acc.test.mean and learner.id and task.id
Friedman chi-squared = 2, df = 1, p-value = 0.1573

Warning message:
In friedmanPostHocTestBMR(bmr, p.value = 0.05) :
  Cannot reject null hypothesis of overall Friedman test,
             returning overall Friedman test.

mlr documentation built on April 1, 2018, 12:03 p.m.