compareModels: Compare accuracy of alternative classification methods

Description Usage Arguments Details Value Note Author(s) Examples

Description

Compare, between models, probabilities that the models assign to membership in the correct group or class. Probabilites should be estimated from cross-validation or from bootstrap out-of-bag data or preferably for test data that are completely separate from the data used to dervive the model.

Usage

1
2
compareModels(groups, estprobs = list(lda = NULL, rf = NULL),
            gpnames = NULL, robust = TRUE, print = TRUE)

Arguments

groups

Factor that specifies the groups

estprobs

List whose elements (with names that identify the models) are matrices that give for each observation (row) estimated probabilities of membership for each of the groups (columns).

gpnames

Character: names for groups, if different from levels(groups)

robust

Logical, TRUE or FALSE

print

Logical. Should results be printed?

Details

The estimated probabilities are compared directly, under normal distribution assumptions. An effect is fitted for each observation, plus an effect for the method. Comparison on a logit scale may sometimes be preferable. An option to allow this is scheduled for incorporation in a later version.

Value

modelAVS

Average accuracies for models

modelSE

Approximate average SE for comparing models

gpAVS

Average accuracies for groups

gpSE

Approximate average SE for comparing groups

obsEff

Effects assigned to individual observations

Note

The analysis estimates effects due to model and group (gp), after accounting for differences between observations.

Author(s)

John Maindonald

Examples

1
2
3
4
5
6
7
8
9
library(MASS)
library(DAAG)
library(randomForest)
ldahat <- lda(species ~ length+breadth, data=cuckoos, CV=TRUE)$posterior
qdahat <- qda(species ~ length+breadth, data=cuckoos, CV=TRUE)$posterior
rfhat <- predict(randomForest(species ~ length+breadth, data=cuckoos),
                 type="prob")
compareModels(groups=cuckoos$species, estprobs=list(lda=ldahat,
              qda=qdahat, rf=rfhat), robust=FALSE)

Example output

Loading required package: lattice

Attaching package: 'DAAG'

The following object is masked from 'package:MASS':

    hills

randomForest 4.6-14
Type rfNews() to see new features/changes/bug fixes.
[1] "Average accuracies for groups:"
gphedge.sparrow  gpmeadow.pipit  gppied.wagtail         gprobin    gptree.pipit 
         0.1370          0.5257          0.1328          0.1458          0.1379 
         gpwren 
         0.5464 
Approx sed 
    0.0253 
[1] "Average accuracies for methods:"
   lda    qda     rf 
0.3260 0.3385 0.3396 
Approx sed 
    0.0126 

gamclass documentation built on Nov. 14, 2020, 1:07 a.m.