Description Usage Arguments Details Value References See Also Examples
View source: R/relativeOverfitting.R
Estimates the relative overfitting of a model as the ratio of the difference in test and train performance to the difference of test performance in the no-information case and train performance. In the no-information case the features carry no information with respect to the prediction. This is simulated by permuting features and predictions.
1 2 | estimateRelativeOverfitting(predish, measures, task, learner = NULL,
pred.train = NULL, iter = 1)
|
predish |
[ |
measures |
[ |
task |
[ |
learner |
[ |
pred.train |
[ |
iter |
[ |
Currently only support for classification and regression tasks is implemented.
[data.frame
]. Relative overfitting estimate(s), named by measure(s), for each resampling iteration.
Bradley Efron and Robert Tibshirani; Improvements on Cross-Validation: The .632+ Bootstrap Method, Journal of the American Statistical Association, Vol. 92, No. 438. (Jun., 1997), pp. 548-560.
Other performance: ConfusionMatrix
,
calculateConfusionMatrix
,
calculateROCMeasures
,
makeCostMeasure
,
makeCustomResampledMeasure
,
makeMeasure
, measures
,
performance
, setAggregation
,
setMeasurePars
1 2 3 4 5 6 | task = makeClassifTask(data = iris, target = "Species")
rdesc = makeResampleDesc("CV", iters = 2)
estimateRelativeOverfitting(rdesc, acc, task, makeLearner("classif.knn"))
estimateRelativeOverfitting(rdesc, acc, task, makeLearner("classif.lda"))
rpred = resample("classif.knn", task, rdesc)$pred
estimateRelativeOverfitting(rpred, acc, task)
|
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.