compareModels: Compare accuracy of alternative classification methods
In DAAGxtras: Data Sets and Functions, supplementary to DAAG

Description Usage Arguments Details Value Note Author(s) Examples

Compare, between models, probabilities that the models assign to membership in the correct group or class. Probabilites should be estimated from cross-validation or from bootstrap out-of-bag data or preferably for test data that are completely separate from the data used to dervive the model.

1 2	compareModels(groups = fgl$type, estprobs = list(lda = NULL, rf = NULL), gpnames = NULL, robust = TRUE, print = TRUE)

`groups`	Factor that specifies the groups
`estprobs`	List whose elements (with names that identify the models) are matrices that give for each observation (row) estimated probabilities of membership for each of the groups (columns).
`gpnames`	Character: names for groups, if different from `levels(groups)`
`robust`	Logical, `TRUE` or `FALSE`
`print`	Logical. Should results be printed?

The estimated probabilities are compared directly, under normal distribution assumptions. An effect is fitted for each observation, plus an effect for the method. Comparison on a logit scale may sometimes be preferable. An option to allow this is scheduled for incorporation in a later version.

`modelAVS`	Average accuracies for models
`modelSE`	Approximate average SE for comparing models
`gpAVS`	Average accuracies for groups
`gpSE`	Approximate average SE for comparing groups
`obsEff`	Effects assigned to individual observations

The analysis estimates effects due to model and group (gp), after accounting for differences between observations.

John Maindonald

library(MASS)
library(DAAG)
ldahat <- lda(species ~ length+breadth, data=cuckoos, CV=TRUE)$posterior
qdahat <- qda(species ~ length+breadth, data=cuckoos, CV=TRUE)$posterior
compareModels(groups=cuckoos$species, estprobs=list(lda=ldahat,
              qda=qdahat), robust=FALSE)
rfOUT <- try(require(randomForest, quietly=TRUE))
rfOUT.log <- is.logical(rfOUT)
if ((rfOUT.log==TRUE)&(rfOUT==TRUE)){
rfhat <- predict(randomForest(species ~ length+breadth, data=cuckoos),
                 type="prob")
compareModels(groups=cuckoos$species, estprobs=list(lda=ldahat,
              qda=qdahat, rf=rfhat), robust=FALSE)
}