fscoreSummary: Precision-Recall-Fscore performance measures across resamples

In luca-scr/smef: Statistical Methods for Economy and Finance

Description

Calculates information retrieval measures (Precision, Recall, F1, F2) for evaluating a two-class classifier output quality. This function can be used in train() function for selecting the hyperparameter(s) of a classifier. This can be achieved by specifying the argument metric in train() function call, and summaryFunction = fscoreSummary and classProbs = TRUE in trainControl. See examples below.

Usage

fscoreSummary(data, lev = NULL, model = NULL, ...)

Arguments

data	a data frame with columns obs and pred for the observed and predicted outcomes, and prob for predicted probabilities for each class. See the classProbs argument to trainControl.
lev	a character vector of factors levels for the response.
model	a character string for the model name (as taken from the method argument of train.
...	additional arguments to be passed to the low level functions.

Value

A vector containing the metrics.

See Also

twoClassSummary, ceSummary

Examples

data = caret::twoClassSim(200)
describe(data)

mod1 = train(Class ~ . , data = data,
             method = "rpart2",
             tuneGrid = expand.grid(maxdepth = 1:10),
             metric = "F1",
             trControl = trainControl(method = "cv", number = 10,
                                      classProbs = TRUE,
                                      savePredictions = TRUE,
                                      summaryFunction = fscoreSummary,
                                      selectionFunction = "best") )
mod1
plot(mod1)

mod2 = train(Class ~ . , data = data,
             method = "rpart2",
             tuneGrid = expand.grid(maxdepth = 1:10),
             metric = "F2",
             trControl = trainControl(method = "cv", number = 10,
                                      classProbs = TRUE,
                                      summaryFunction = fscoreSummary,
                                      selectionFunction = "best") )
mod2
plot(mod2)

mod3 = train(Class ~ . , data = data,
             method = "rf",
             tuneGrid = expand.grid(mtry = 1:5),
             metric = "F1",
             trControl = trainControl(method = "cv", number = 10,
                                      classProbs = TRUE,
                                      summaryFunction = fscoreSummary,
                                      selectionFunction = "best") )
mod3
plot(mod3)

mod4 = train(Class ~ . , data = data,
             method = "rf",
             tuneGrid = expand.grid(mtry = 1:5),
             metric = "F2",
             trControl = trainControl(method = "cv", number = 10,
                                      classProbs = TRUE,
                                      summaryFunction = fscoreSummary,
                                      selectionFunction = "best") )
mod4
plot(mod4)

luca-scr/smef documentation built on Feb. 22, 2022, 8:32 p.m.