score_binary: Score predictions for binary events

In pminternal: Internal Validation of Clinical Prediction Models

Score predictions for binary events

Description

Calculate scores summarizing discrimination/calibration of predictions against observed binary events. If score_fun is not defined when calling validate this function is used.

Usage

score_binary(y, p, ...)

Arguments

y	vector containing a binary outcome
p	vector of predictions
...	additional arguments. This function only supports calib_args as an optional argument. calib_args should contain arguments for pmcalibration::pmcalibration. If a calibration plot (apparent vs bias corrected calibration curves via cal_plot) is desired the argument 'eval' should be provided. This should be the points at which to evaluate the calibration curve on each boot resample or crossvalidation fold. A good option would be calib_args = list(eval = seq(min(p), max(p), length.out=100)); where p are predictions from the original model evaluated on the original data. Dots can be used to supply additional arguments to user-defined functions.

Details

The following measures are returned in a named vector.

C

the c-statistic (aka area under the ROC curve). Probability that randomly selected observation with y = 1 with have higher p compared to randomly selected y = 0.

Brier

mean squared error - mean((y - p)^2)

Intercept

Intercept from a logistic calibration model: glm(y ~ 1 + offset(qlogis(p)), family="binomial")

Slope

Slope from a logistic calibration model: glm(y ~ 1 + qlogis(p), family="binomial")

Eavg

average absolute difference between p and calibration curve (aka integrated calibration index or ICI).

E50

median absolute difference between p and calibration curve

E90

90th percentile absolute difference between p and calibration curve

Emax

maximum absolute difference between p and calibration curve

ECI

average squared difference between p and calibration curve. Estimated calibration index (Van Hoorde et al. 2015)

cal_plot

if eval is specified (via calib_args), values for plotting apparent and bias-corrected calibration curves are returned (see cal_plot). By default these are omitted from the summary printed (see summary.internal_validate).

Logistic calibration and other calibration metrics from non-linear calibration curves assessing 'moderate-calibration' (Eavg, E50, E90, Emax, ECI; see references) are calculated via the pmcalibration package. The default settings can be modified by passing calib_args to validate call. calib_args should be a named list corresponding to arguments to pmcalibration::pmcalibration.

Value

a named vector of scores (see Details)

References

Austin PC, Steyerberg EW. (2019) The Integrated Calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models. Statistics in Medicine. 38, pp. 1–15. https://doi.org/10.1002/sim.8281

Van Hoorde, K., Van Huffel, S., Timmerman, D., Bourne, T., Van Calster, B. (2015). A spline-based tool to assess and visualize the calibration of multiclass risk predictions. Journal of Biomedical Informatics, 54, pp. 283-93

Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina M., Steyerberg E.W. (2016). A calibration hierarchy for risk models was defined: from utopia to empirical data. Journal of Clinical Epidemiology, 74, pp. 167-176

Examples

p <- runif(100)
y <- rbinom(length(p), 1, p)
score_binary(y = y, p = p)

pminternal documentation built on April 4, 2025, 3:30 a.m.