Performance estimation functions

Share:

Description

These functions determine the performance of fitted model based on its predictions. They are used both for evaluating whole modeling procedures and to tune model parameters, i.e. find the parameter values with the best performance. The parameter tuning routine is designed to minimize its error function (or optimization criteria), which is why functions that are to be maximized must have their sign changed, like neg_auc.

Usage

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
error_rate(truth, prediction, allow_rejection = !missing(rejection_cost),
  rejection_cost)

neg_auc(truth, prediction)

rmse(truth, prediction, na.rm = FALSE)

mse(truth, prediction, na.rm = FALSE)

neg_harrell_c(truth, prediction, na.rm = FALSE)

Arguments

truth

The true response values, be it class labels, numeric values or survival outcomes.

prediction

A prediction object.

allow_rejection

If FALSE missing prediction values will produce an error. If TRUE missing values will be given a cost specified by the rejection_cost argument.

rejection_cost

See the argument allow_rejection. If missing a rejection cost equivalent to the error rate obtained when assigning all test observations to the most common class will be used.

na.rm

Whether to remove missing values or not.

Details

Custom performance estimation functions should be implemented as follows:

function(truth, prediction)

truth

A vector of true responses.

prediction

Prediction returned from the prediction function.

In most cases the true response and the predictions are of the same type, e.g. true and fitted values in a regression or class labels in a classification problem, but it is not a requirement. An example of different types could be if the prediction function produce class probabilities for all classes rather than one label, or the risks that the observations will experience the event of interest, to be compared to the actual outcome that it did occur or has not yet occurred at a specific time point. See neg_harrell_c for an example of the latter.

Author(s)

Christofer B├Ącklin

See Also

emil, neg_gmpa, modeling_procedure, extension

Want to suggest features or report bugs for rdrr.io? Use the GitHub issue tracker.