comperf: Compute model performance

In gbts: Hyperparameter Search for Gradient Boosted Trees

Description

This function computes model performance given a vector of response values and a vector of predictions.

Usage

comperf(y, yhat, w = rep(1, length(y)), pfmc = NULL, cdfx = "fpr",
  cdfy = "tpr", dspt = 0.5)

Arguments

y	a vector of numeric response values.
yhat	a vector of model predictions.
w	an optional vector of observation weights.
pfmc	a character of the performance metric. For binary classification, pfmc accepts: "acc": accuracy "dev": deviance "ks": Kolmogorov-Smirnov (KS) statistic "auc": area under the ROC curve. Use the cdfx and cdfy arguments to specify the cumulative distributions for the x-axis and y-axis of the ROC curve, respectively. The default ROC curve is given by true positive rate (y-axis) vs. false positive rate (x-axis). "roc": rate on the y-axis of the ROC curve at a particular decision point (threshold) on the x-axis specified by the dspt argument. For example, if the desired performance metric is true positive rate at the 5% false positive rate, specify pfmc="roc", cdfx="fpr", cdfy="tpr", and dspt=0.05. For regression, pfmc accepts: "mse": mean squared error "mae": mean absolute error "rsq": r-squared (coefficient of determination)
cdfx	a character of the cumulative distribution for the x-axis. Supported values are "fpr": false positive rate "fnr": false negative rate "rpp": rate of positive prediction
cdfy	a character of the cumulative distribution for the y-axis. Supported values are "tpr": true positive rate "tnr": true negative rate
dspt	a decision point (threshold) in [0, 1] for binary classification. If pfmc="acc", instances with probabilities <= dspt are predicted as negative, and those with probabilities > dspt are predicted as positive. If pfmc="roc", dspt is a threhold on the x-axis of the ROC curve such that the corresponding value on the y-axis is used as the performance metric. For example, if the desired performance metric is the true positive rate at the 5% false positive rate, specify pfmc="roc", cdfx="fpr", cdfy="tpr", and dspt=0.05.

Value

A single or a vector of numeric values of model performance, or a list of two components x and y representing the ROC curve.

Author(s)

Waley W. J. Liang <[e-mail]>

See Also

gbts, predict.gbts

Examples

y = c(0, 1, 0, 1, 1, 1)
yhat = c(0.5, 0.9, 0.2, 0.7, 0.6,  0.4)
comperf(y, yhat, pfmc = "auc")
# 0.875

y = 1:10
yhat = c(1:5 - 0.1, 6:10 + 0.1)
comperf(y, yhat, pfmc = "mse")
# 0.01

gbts documentation built on May 2, 2019, 9:42 a.m.