assess: Accuracy measures

In camroach87/validatr:

Description

Returns prediction accuracy measures.

Usage

assess(object)

Arguments

object

a validatr object containing cross-validation folds and predictions.

Details

Accuracy measures returned for regression include:

• Absolute error (AE)

• Mean absolute error (MAE)

• Mean absolute percentage error (MAPE)

• Root mean square error (RMSE)

• Symmetric mean absolute percentage error (SMAPE)

Time-series accuracy measures include the above measures plus:

• Mean absolute scaled error (MASE)

All regression and time-series accuracy measures except for SMAPE are defined as in the paper Hyndman, Rob J., and Anne B. Koehler. 2006. “Another Look at Measures of Forecast Accuracy.” International Journal of Forecasting 22 (4): 679–88. SMAPE is defined similarly, but with absolute values in the denominator to ensure measures fall between 0 and 200.

Classification accuracy measures include:

• Accuracy

• Precision

• Sensitivity

• Specificity

• F-score

These measures are defined as in Sokolova, Marina, and Guy Lapalme. 2009. “A Systematic Analysis of Performance Measures for Classification Tasks.” Information Processing & Management 45 (4): 427–37. For multi-class classification problems, macro-averaging is used to ensure large classes are not favoured. Macro-averaging averages the performance of each class. Most of these are defined in Table 3 of the paper. Since the multi-class measures do not reduce to the binary measures when the number of classes is equal to two the binary classification accuracy measures in Table 2 have also been included and are activated when the response variable in the input data is Boolean.

Value

A data frame with the accuracy measures listed above.

Examples

iris %>%
  validatr(Sepal.Length, k = 3) %>%
  model(Model1 = lm(Sepal.Length ~ ., data = train),
        Model2 = lm(Sepal.Length ~ Sepal.Width + Petal.Width, data = train)) %>%
  predict(Model1 = predict(Model1, newdata = validation),
          Model2 = predict(Model2, newdata = validation)) %>%
  assess()

camroach87/validatr documentation built on May 14, 2019, 2:41 p.m.