predict_metrics: Prediction Metrics

In jashu/beset: Best Subset Predictive Modeling

Prediction Metrics

Description

Calculate deviance, mean absolute error, mean cross entropy, mean squared error, and R-squared (as fraction of deviance explained) to measure how well a generalized linear model predicts test data.

Usage

predict_metrics(object, test_data)

predict_metrics_(y, y_hat, family, mu = NULL, phi = NULL, theta = NULL)

Arguments

object	A model object of class "glm", "negbin", or "zeroinfl".
test_data	A data frame containing new data for the response and all predictors that were used to fit the model object.
y	Vector of actual observed values
y_hat	Vector of predicted values.
family	Character string giving the name of the error distribution for y_hat. Currently supported distributions are "gaussian", "binomial", "poisson", "negbin" (negative binomial), "zip" (zero-inflated Poisson), and "zinb" (zero-inflated negative binomial).
mu	(Required for zero-inflated models) the predicted values for the count component of the model.
phi	(Required for zero-inflated models) the predicted values for the zero component of the model.
theta	(Required for negative binomial models) the estimated dispersion parameter theta.

Details

predict_metrics uses a generalized linear model previously fit to a training set to predict responses for a test set. It then computes the residual sum of squares and the log-likelihood of the predicted responses, as well as the log-likelihoods for the corresponding saturated model (a model with one free parameter per observation) and null model (a model with only an intercept) fit to the true responses. These quantities are used to derive the metrics described below.

predict_metrics_ is the workhorse function: there is no need to call this directly but is more efficient for programming if the actual response vector, the predicted response vector(s), and dispersion parameter (if applicable) have already been calculated.

Value

For binomial models, a list giving the area under the ROC curve ("auc"), mean cross entropy AKA log loss ("mce"), mean squared error AKA Brier score ("mse"), and deviance R-squared ("rsq'). For all other models, mean absolute error ("mae") takes the place of "auc" but otherwise the metrics are the same.

Mean squared error (MSE) and mean cross entropy (MCE)

MSE is the average of the squared difference between each predicted response ŷ and the actual observed response y. MCE is an analagous information-theoretic quantity that averages the negative logs of the probability density functions for ŷ evaluated at y.

Note that cross entropy simply parameterizes prediction error in terms of relative likelihood; if applied to the training set instead of the test set, MCE equals the negative log-likelihood of the model divided by the number of observations. (For logistic regression, this is also known as "log-loss".) Given that GLMs minimize the negative log-likelihood, cross entropy defines a unified loss function for both fitting and cross-validating GLMs.

R-squared and deviance explained

The prediction R-squared provides a metric analagous to R-squared, except instead of describing how well a model fits the original training data, it describes how well the model predicts independent test data. Instead of using the traditional formula, 1 - RSS / TSS, where RSS corresponds to the sum of the squared residuals, (y - ŷ)^2, and TSS is the total sum of squares, it is calculated as 1 - dev_PRED / dev_NULL, where dev_PRED is the deviance for the model prediction and dev_NULL is the deviance for the null model. This yields a quantity equivalent to the traditional R-squared formula for linear models with normal error distributions, but a different quantity for exponential family regression models (e.g., logistic regression) that preserves the interpretation of R-squared as the fraction of uncertainty explained (Cameron & Windmeijer, 1997). (See r2d for more details.)

Note that the prediction R-squared can be negative if overfitting is severe; i.e., when predicting new data, the model performs worse than if one were to always predict the mean response.

Supported Probability Distributions

Currently the following "families" are supported: Gaussian, binomial, Poisson, negative binomial, zero-inflated Poisson, and zero-inflated negative binomial.

See Also

r2d, logLik

jashu/beset documentation built on April 20, 2023, 5:28 a.m.