experimentalComparison: Carry out Experimental Comparisons Among Learning Systems
In DMwR: Functions and data for "Data Mining with R"

Description Usage Arguments Details Value Author(s) References See Also Examples

This function can be used to carry out different types of experimental comparisons among learning systems on a set of predictive tasks. This is a generic function that should work with any learning system provided a few assumptions are met. The function implements different experimental methodologies, namely: cross validation, leave one out cross validation, hold-out, monte carlo simulations and bootstrap.

1	experimentalComparison(datasets, systems, setts, ...)

`datasets`	This is a list of objects of class `dataset`, containing the data sets that will be used in the comparison.
`systems`	This is a list of objects of class `learner`, containing the learning systems that will be used in the comparison.
`setts`	This is an object belonging to any of the sub-classes of the virtual class `expSettings`. It is the class of this object that determines the type of experimental comparison that will be carried out. See section Details for the possible values.
`...`	Other parameter settings that are to be passed to the functions actually carrying out the experiments (e.g. `crossValidation`, etc.).

The goal of this function is to allow to carry out different types of experimental comparisons between a set of learning systems over a set of predictive tasks. The idea is that all learning system will be compared over the same data partitions for each of the tasks thus ensuring fare comparisons and also allowing for proper statistical tests of significance of the observed differences, to be carried out.

Currently, the function allows for 5 different types of experimental comparisons to be carried out. These different types are in effect, different estimation methods for the target evaluation statistics that are to be used in evaluation the different learners over the tasks. The method to be used is determined by the class of the object provided in the argument setts. The following are the possibilities:

"Cross validation": this type of estimates can be obtained by providing in the setts argument and object of class cvSettings. More details on this type of experiments can be obtained in the help page of the function crossValidation.

"Leave one out cross validation": this type of estimates can be obtained by providing in the setts argument and object of class loocvSettings. More details on this type of experiments can be obtained in the help page of the function loocv.

"Hold out": this type of estimates can be obtained by providing in the setts argument and object of class hldSettings. More details on this type of experiments can be obtained in the help page of the function holdOut.

"Monte carlo": this type of estimates can be obtained by providing in the setts argument and object of class mcSettings. More details on this type of experiments can be obtained in the help page of the function monteCarlo.

"Bootstrap": this type of estimates can be obtained by providing in the setts argument and object of class bootSettings. More details on this type of experiments can be obtained in the help page of the function bootstrap.

The result of the function is an object of class compExp (type "class?compExp" for details).

Luis Torgo ltorgo@dcc.fc.up.pt

Torgo, L. (2010) Data Mining using R: learning with case studies, CRC Press (ISBN: 9781439810187).

http://www.dcc.fc.up.pt/~ltorgo/DataMiningWithR

variants,bestScores,rankSystems, compAnalysis, crossValidation, loocv, holdOut, monteCarlo, bootstrap, compExp, cvSettings, hldSettings, mcSettings, loocvSettings, bootSettings

## Estimating several evaluation metrics on different variants of a
## regression tree and of a SVM, on  two data sets, using one repetition
## of 10-fold CV
data(swiss)
data(mtcars)

## First the user defined functions 
cv.rpartXse <- function(form, train, test, ...) {
    require(DMwR)
    t <- rpartXse(form, train, ...)
    p <- predict(t, test)
    mse <- mean((p - resp(form, test))^2)
    c(nmse = mse/mean((mean(resp(form, train)) - resp(form, test))^2), 
        mse = mse)
}

results <- experimentalComparison(
               c(dataset(Infant.Mortality ~ ., swiss),
                 dataset(mpg ~ ., mtcars)),
               c(variants('cv.rpartXse',se=c(0,0.5,1))),
               cvSettings(1,10,1234)
                                 )
## Check a summary of the results
summary(results)

bestScores(results)

## Check the statistical significance against the best model in terms of
## nmse for the swiss data set
compAnalysis(results,against='cv.rpartXse.v3',stats='nmse',datasets='swiss')

## Plot them
## Not run: 
plot(results)

## End(Not run)