estimateRelativeOverfitting: Estimate relative overfitting.

In shuodata/mlr-master: Machine Learning in R

Description

Estimates the relative overfitting of a model as the ratio of the difference in test and train performance to the difference of test performance in the no-information case and train performance. In the no-information case the features carry no information with respect to the prediction. This is simulated by permuting features and predictions.

Usage

estimateRelativeOverfitting(rdesc, measures, task, learner)

## S3 method for class 'ResampleDesc'
estimateRelativeOverfitting(rdesc, measures, task,
  learner)

Arguments

rdesc	[ResampleDesc] Resampling strategy.
measures	[Measure | list of Measure] Performance measure(s) to evaluate. Default is the default measure for the task, see here getDefaultMeasure.
task	[Task] The task.
learner	[Learner | character(1)] The learner. If you pass a string the learner will be created via makeLearner.

Details

Currently only support for classification and regression tasks is implemented.

Value

[data.frame]. Relative overfitting estimate(s), named by measure(s), for each resampling iteration.

References

Bradley Efron and Robert Tibshirani; Improvements on Cross-Validation: The .632+ Bootstrap Method, Journal of the American Statistical Association, Vol. 92, No. 438. (Jun., 1997), pp. 548-560.

See Also

Other performance: ConfusionMatrix, calculateConfusionMatrix, calculateROCMeasures, makeCostMeasure, makeCustomResampledMeasure, makeMeasure, measures, performance

Examples

task = makeClassifTask(data = iris, target = "Species")
rdesc = makeResampleDesc("CV", iters = 2)
estimateRelativeOverfitting(rdesc, acc, task, makeLearner("classif.knn"))
estimateRelativeOverfitting(rdesc, acc, task, makeLearner("classif.lda"))

shuodata/mlr-master documentation built on May 20, 2019, 3:33 p.m.