single_mod_results: Compute Performance Metrics of a Single Caret Model Object...

In AndrewKostandy/MLtoolkit: Functions to Help with Machine Learning & Feature Engineering Tasks

Description

Compute various types of performance metrics across resamples for a binary classification or regression model from the caret package.

The metrics computed for binary classification are: Area under ROC Curve (AUROC), Sensitivity, Specificity, Area under Precision-Recall Curve (AUPRC), Precision, F1 Score, Accuracy, Cohen's Kappa, Log Loss, Matthews Correlation Coefficient, Concordance, Discordance, Somer's D, KS Statistic, and False Discovery Rate.

The metrics computed for regression are: Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), Spearman's Rho, Concordance Correlation Coefficient, and RSquared. Note that MAPE will not be provided if any observations equal zero to avoid division by zero.

Note that the savePredictions argument in caret's trainControl function should be set to "final" so that the resampling results of the model with the optimal tuning parameters are available: trainControl(..., savePredictions = "final")

Usage

single_mod_results(mod_object, mod_name)

Arguments

mod_object	A caret model object.
mod_name	A name for the model.

Value

Returns a dataframe with the the performance metrics of the model object across resamples.

Author(s)

Andrew Kostandy ([e-mail])

References

The "InformationValue", "caret", and "MLmetrics" packages were used to compute many of the metrics.

Examples

train_ctrl <- trainControl(method = "repeatedcv",
                           number = 10,
                           repeats = 4,
                           summaryFunction = defaultSummary,
                           savePredictions = "final")

lm_fit_1 <- train(Sepal.Length ~ Sepal.Width, data = iris,
                  method = "lm",
                  metric = "RMSE",
                  trControl = train_ctrl)

single_mod_results(lm_fit_1, "LM 1")

AndrewKostandy/MLtoolkit documentation built on May 7, 2019, 9:51 p.m.