ceSummary: Classification error and Brier score performance measures...

In luca-scr/smef: Statistical Methods for Economy and Finance

Description

Calculates classification error and Brier score measures for evaluating a multi-class classifier output quality. This function can be used in train() function for selecting the hyperparameter(s) of a classifier. This can be achieved by specifying the argument metric in train() function call, and summaryFunction = ceSummary and classProbs = TRUE in trainControl. See examples below.

Usage

ceSummary(data, lev = NULL, model = NULL, ...)

Arguments

data	a data frame with columns obs and pred for the observed and predicted outcomes, and prob for predicted probabilities for each class. See the classProbs argument to trainControl.
lev	a character vector of factors levels for the response.
model	a character string for the model name (as taken from the method argument of train.
...	additional arguments to be passed to the low level functions.

Value

A vector containing the metrics.

See Also

twoClassSummary, fscoreSummary

Examples

data = mlbench::mlbench.waveform(300)
data = data.frame(Class = data$classes, data$x)
data$Class = paste0("Class", data$Class)
describe(data)

mod1 = train(Class ~ . , data = data,
             method = "rpart2",
             tuneGrid = expand.grid(maxdepth = 1:10),
             metric = "ClassError",
             maximize = FALSE,
             trControl = trainControl(method = "cv", number = 10,
                                      classProbs = TRUE,
                                      summaryFunction = ceSummary,
                                      selectionFunction = "best") )
mod1
ggplot(mod1) +
  scale_x_continuous(breaks = mod1$results$maxdepth) +
  geom_errorbar(aes(ymin = with(mod1$results, ClassError - ClassErrorSD/sqrt(10)),
                    ymax = with(mod1$results, ClassError + ClassErrorSD/sqrt(10))),
                width = 0.3)

mod2 = train(Class ~ . , data = data,
             method = "rpart2",
             tuneGrid = expand.grid(maxdepth = 1:10),
             metric = "BrierScore",
             maximize = FALSE,
             trControl = trainControl(method = "cv", number = 10,
                                      classProbs = TRUE,
                                      summaryFunction = ceSummary,
                                      selectionFunction = "best") )
mod2
ggplot(mod2) +
  scale_x_continuous(breaks = mod2$results$maxdepth) +
  geom_errorbar(aes(ymin = with(mod2$results, BrierScore - BrierScoreSD/sqrt(10)),
                    ymax = with(mod2$results, BrierScore + BrierScoreSD/sqrt(10))),
                width = 0.3)

luca-scr/smef documentation built on Feb. 22, 2022, 8:32 p.m.