summary.predx: Fit statistic summary .

In PiotrTymoszuk/caretExtra: Extra functionality for predictions and quality control of caret models

Fit statistic summary .

Description

Calculates fit statistics for models (caretx) and predictions (predx). For regression, those include the fit errors and outcome - fitted value correlations. For binary classification, concordance (C-index), kappa, accuracy, receiver-operator characteristic (ROC), weighted average class probabilities and Brier scores are returned. For multiple-class predictions and models, accuracy, kappa, weighted average class probabilities and Brier scores are returned.

Usage

## S3 method for class 'predx'
summary(object, ci_method = c("percentile", "bca", "norm"), ...)

## S3 method for class 'caretx'
summary(
  object,
  newdata = NULL,
  ci_method = c("percentile", "bca", "norm"),
  plain = FALSE,
  ...
)

Arguments

object	a caretx model or predx prediction object.
ci_method	method used for calculation of 95\ normal distribution, percentile or BCA (bca). Defaults to 'percentile'.
...	extra arguments, currently none.
newdata	optional, a data frame with the test data.
plain	logical, should the output be coerced to a single data frame?

Details

Harrell's C-index is calculated as specified for concordance. Receiver-operating characteristic (ROC), accuracy and unweighted Cohen's kappa is computed for prediction probabilities using caret's multiClassSummary. Fit errors are based on working residuals and include mean absolute error (MAE), mean squared error (MSE) and root-MSE (RMSE). Pseudo R squared is calculated as 1 - MSE/Var(y). Pearson correlation is obtained with cor.test, Spearman correlation is computed with SpearmanRho, Kendall's TauB is obtained with cor. For cross-validation (CV) prediction, statistic values are calculated as mean across the CV with 95\ For multi-class predictions and models, statistics referring to discrimination of single classes like sensitivity or recall are averaged over all classes.

Value

a data frame with the fit summary statistic (predx) or a list of such data frames for the training, resample (CV) and test data (caretx).

References

Kuhn M. Building predictive models in R using the caret package. J Stat Softw (2008) 28:1–26. doi:10.18637/jss.v028.i05

Brier GW. VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY. Mon Weather Rev (1950) 78:1–3. doi:10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2

Goldstein-Greenwood J. A Brief on Brier Scores | UVA Library. (2021) Available at: https://library.virginia.edu/data/articles/a-brief-on-brier-scores

Cohen J. A Coefficient of Agreement for Nominal Scales. Educ Psychol Meas (1960) 20:37–46. doi:10.1177/001316446002000104

PiotrTymoszuk/caretExtra documentation built on Oct. 15, 2023, 10:03 p.m.