modelAccs: Generate accuracy statistics and variable importance for a...

In henkelstone/NPEL.Classification: Classifaction and data-handling routines for NPEL Caribou Project

Description

Given a list of models, this function runs classAcc and npelVIMP on each. See details for a description of how the data is encapsulated for return.

Usage

modelAccs(models, VIMP = TRUE, ...)

Arguments

models	is either a list of model objects on which to find error statistics, or a single model to evaluate.
VIMP	(optional) should we attempt to append the VIMP data; defaults to TRUE
...	(optional) other parameters to pass to classAcc and/or npelVIMP.

Details

The function returns a named list of the various accuracy and variable importance metrics:

• Accuracies contains the complete names list returned by classAcc; see the help on that method for more information on that data structure.

• VIMP a named list containing the variable importance matrices (VIMP) for each included model. Each entry in the list is a data frame in which columns are different classes and rows are different input variables; specific entries represent the variable importance of a given input variable on a particular class.

• VIMPoverall a data.frame showing the overall variable importance (VIMP) for a given model.

  Columns are different models and entries in the table are the standardized VIMP for an input variable on that model. Values have been standardized by column so they are (somewhat) comparable as the algorithm by which VIMP is computed varies by model–hence the largest value will be 1 in every column. This makes the columns approximately comparable, however, values should be taken with a grain of salt, and perhaps rank order is the most robust comparison.

Value

This function returns different values depending on whether the model is categorical or continuous. For categorical data it returns a named list of accuracy and VIMP statistics:

• confMatrix = confusion matrix

• userAcc = user accuracy

• prodAcc = producer accuracy

• overallAcc = overall accuracy

• kappa = kappa

• VIMP = VIMP matrices

• VIMPoverall = overall VIMP

For continuous data:

• overallAcc = overall r-squared

• mse = mean squared error

Note

See the details section of npelVIMP for a discussion of the limitations of our VIMP metric.

See Also

classAcc for more on continuous and categorical accuracies, and npelVIMP for more on how that metric is computed and it's limitations. Also see modelsValid for a function to compute accuracies using an independent dataset.

Examples

# Categorical Data
data ('siteData')
modelRun <- generateModels (data = siteData,
                            modelTypes = suppModels,
                            x = c('brtns','grnns','wetns','dem','slp','asp','hsd'),
                            y = 'ecoType',
                            grouping = ecoGroup[['domSpecies','transform']],
                            echo = FALSE)
mE <- modelAccs (modelRun, ecoGroup[['domSpecies','labels']])
str(mE,1)

# Continuous Data
modelRun <- generateModels (data = siteData,
                            modelTypes = contModels,
                            x = c('brtns','grnns','wetns','dem','slp','asp','hsd'),
                            y = 'easting',
                            echo = FALSE)
mE <- modelAccs (modelRun)
str(mE,1)

henkelstone/NPEL.Classification documentation built on May 17, 2019, 3:42 p.m.