Evaluation of learner accuracy by ROSE
Description
Given a classifier and a set of data, this function exploits ROSE generation of synthetic samples to provide holdout, bootstrap or leaveKout crossvalidation estimates of a specified accuracy measure.
Usage
1 2 3 4 5 6 
Arguments
formula 
An object of class 
data 
An optional data frame, list or environment (or object
coercible to a data frame by 
learner 
Either a builtin R or an user defined function that fits a classifier and that returns a vector of predicted values. See “Details” below. 
acc.measure 
One among 
extr.pred 
An optional function that extracts from the output of a 
method.assess 
One among 
K 
An integer value indicating the size of the subsets created when

B 
The number of bootstrap replications to set when 
control.learner 
Further arguments to be passed to 
control.rose 
Optional arguments to be passed to 
control.predict 
Further arguments to be passed to 
control.accuracy 
Optional arguments to be passed to either 
trace 
logical, if 
subset 
An optional vector specifying a subset of observations to be used in the sampling and learning process.
The default is set by the 
na.action 
A function which indicates what should happen when the data contain 'NA's.
The default is set by the 
seed 
A single value, interpreted as an integer, recommended to specify seeds and keep trace of the generated ROSE sample/es. 
Details
This function estimates a measure of accuracy of a classifier specified by the user by using either holdout, crossvalidation, or bootstrap estimators. Operationally, the classifier is trained over synthetic data generated by ROSE and then evaluated on the original data.
Whatever accuracy measure and estimator are chosen, the true accuracy depends
on the probability distribution underlying the training data. This is clearly affected by the imbalance
and its estimation is then regulated by argument control.rose
.
A default setting of the arguments (that is, p=0.5
) entails the estimation of the learner accuracy
conditional to a balanced training set. In order to estimate the accuracy of a learner fitted on unbalanced data,
the user may set argument p
of control.rose
to the proportion of
positive examples in the observed sample. See Example 2 below and, for further details, Menardi and Torelli (2014).
To the aim of a grater flexibility, ROSE.eval
is not linked to the use of a specific learner and works virtually with any classifier.
The actual implementation supports the following two type of learner
.
In the first case, learner
has a 'standard' behavior in the sense that it is a function having formula
as a mandatory argument and retrieves an object whose class is associated to a predict
method.
The user that is willing to define her/his own learner
must follow the implicit convention that when a classed object is created, then the function name and the class should match (such as lm
, glm
, rpart
, tree
, nnet
, lda
, etc). Furthermore, since predict
returns are very heterogeneous, the user is allowed to define some function extr.pred
which extracts from the output of predict
the desired vector of predicted values.
In the second case, learner
is a wrapper that allows to embed functions that do not meet the aforementioned requirements. The wrapper must have the following mandatory arguments: data
and newdata
, and must return a vector of predicted values. Optional arguments can be passed as well into the wrapper including the ...
and by specifiyng them through control.learner
.
When argument data
in ROSE.eval
is not missing, data
in learner
receives a data frame structured
as the one in input, otherwise it is constructed according to the template provided by formula
.
The same rule applies for argument newdata
with the exception that the class label variable is dropped. See “Examples” below.
Value
The value is an object of class ROSE.eval
which has components
Call 
The matched call. 
method 
The selected method for model assessment. 
measure 
The selected measure to evaluate accuracy. 
acc 
The vector of the estimated measure of accuracy. It has length 1 if 
Note
The function allows the user to include in the formula transformations of predictors or
interactions among them. ROSE samples are generated on the original data and transformations
or interactions are ignored. These are then retrieved in fitting the classifier, provided that
the selected learner function can handle them. See also “Warning” in ROSE
.
References
Lunardon, N., Menardi, G., and Torelli, N. (2014). ROSE: a Package for Binary Imbalanced Learning. R Jorunal, 6:82–92.
Menardi, G. and Torelli, N. (2014). Training and assessing classification rules with imbalanced data. Data Mining and Knowledge Discovery, 28:92–122.
See Also
ROSE
, roc.curve
, accuracy.meas
.
Examples
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149  # 2dimensional data
# loading data
data(hacide)
# in the following examples
# use of a small subset of observations only > argument subset
dat < hacide.train
table(dat$cls)
##Example 1
# classification with logit model
# arguments to glm are passed through control.learner
# leaveoneout crossvalidation estimate of auc of classifier
# trained on balanced data
ROSE.eval(cls~., data=dat, glm, subset=c(1:50, 981:1000),
method.assess="LKOCV", K=5,
control.learner=list(family=binomial), seed=1)
## Not run:
##Example 2
# classification with decision tree
# require package rpart
library(rpart)
# function is needed to extract predicted probability of cls 1
f.pred.rpart < function(x) x[,2]
# holdout estimate of auc of two classifiers
# first classifier trained on ROSE unbalanced sample
# proportion of rare events in original data
p < (table(dat$cls)/sum(table(dat$cls)))[2]
ROSE.eval(cls~., data=dat, rpart, subset=c(1:50, 981:1000),
control.rose=list(p = p), extr.pred=f.pred.rpart, seed=1)
# second classifier trained on ROSE balanced sample
# optional arguments to plot the roc.curve are passed through
# control.accuracy
ROSE.eval(cls~., data=dat, rpart, subset=c(1:50, 981:1000),
control.rose=list(p = 0.5), control.accuracy = list(add.roc = TRUE,
col = 2), extr.pred=f.pred.rpart, seed=1)
##Example 3
# classification with linear discriminant analysis
library(MASS)
# function is needed to extract the predicted values from predict.lda
f.pred.lda < function(z) z$posterior[,2]
# bootstrap estimate of precision of learner trained on balanced data
prec.distr < ROSE.eval(cls~., data=dat, lda, subset=c(1:50, 981:1000),
extr.pred=f.pred.lda, acc.measure="precision",
method.assess="BOOT", B=100, trace=TRUE)
summary(prec.distr)
##Example 4
# compare auc of classification with neural network
# with auc of classification with tree
# require package nnet
# require package tree
library(nnet)
library(tree)
# optional arguments to nnet are passed through control.learner
ROSE.eval(cls~., data=dat, nnet, subset=c(1:50, 981:1000),
method.assess="holdout", control.learn=list(size=1), seed=1)
# optional arguments to plot the roc.curve are passed through
# control.accuracy
# a function is needed to extract predicted probability of class 1
f.pred.rpart < function(x) x[,2]
f.pred.tree < function(x) x[,2]
ROSE.eval(cls~., data=dat, tree, subset=c(1:50, 981:1000),
method.assess="holdout", extr.pred=f.pred.tree,
control.acc=list(add=TRUE, col=2), seed=1)
##Example 5
# An user defined learner with a standard behavior
# Consider a dummy example for illustrative purposes only
# Note that function name and the name of the class returned match
DummyStump < function(formula, ...)
{
mc < match.call()
m < match(c("formula", "data", "na.action", "subset"), names(mc), 0L)
mf < mc[c(1L, m)]
mf[[1L]] < as.name("model.frame")
mf < eval(mf, parent.frame())
data.st < data.frame(mf)
out < list(colname=colnames(data.st)[2], threshold=1)
class(out) < "DummyStump"
out
}
# Associate to DummyStump a predict method
# Usual S3 definition: predic.classname
predict.DummyStump < function(object, newdata)
{
out < newdata[,object$colname]>object$threshold
out
}
ROSE.eval(formula=cls~., data=dat, learner=DummyStump,
subset=c(1:50, 981:1000), method.assess="holdout", seed=3)
##Example 6
# The use of the wrapper for a function with non standard behaviour
# Consider knn in package class
# require package class
library(class)
# the wrapper require two mandatory arguments: data, newdata.
# optional arguments can be passed by including the object '...'
# note that we are going to specify data=data in ROSE.eval
# therefore data in knn.wrap will receive a data set structured
# as dat as well as newdata but with the class label variable dropped
# note that inside the wrapper we dispense to knn
# the needed quantities accordingly
knn.wrap < function(data, newdata, ...)
{
knn(train=data[,1], test=newdata, cl=data[,1], ...)
}
# optional arguments to knn.wrap may be specified in control.learner
ROSE.eval(formula=cls~., data=dat, learner=knn.wrap,
subset=c(1:50, 981:1000), method.assess="holdout",
control.learner=list(k=2, prob=T), seed=1)
# if we swap the columns of dat we have to change the wrapper accordingly
dat < dat[,c("x1","x2","cls")]
# now class label variable is the last one
knn.wrap < function(data, newdata, ...)
{
knn(train=data[,3], test=newdata, cl=data[,3], ...)
}
ROSE.eval(formula=cls~., data=dat, learner=knn.wrap,
subset=c(1:50, 981:1000), method.assess="holdout",
control.learner=list(k=2, prob=T), seed=1)
## End(Not run)
