calibration_slope: Compute performance in terms of various typical calibration...

In srhaile/mscpredmodel: Multiple Score Comparison of Prediction Models with Network Meta-Analysis

Description

Here we provide several functions to compute some typical performance measures for calibration and discrimination (calibration slope, calibration-in-the-large, c-Statistic / AUC, O:E ratio, Brier score, integrated calibration index, and related E50, E90, and Emax). This is however not intended to be an exhaustive set of performance measures.

Usage

calibration_slope(dd, fm)

calibration_large(dd, fm)

c_statistic(dd, fm)

oe_ratio(dd, fm)

brier_score(dd, fm)

int_calib_index(dd, fm)

E50(dd, fm)

E90(dd, fm)

Emax(dd, fm)

Arguments

dd	A dataset.
fm	The formula that will be called by the model, of the form outcome ~ score (character).

Details

Please note that compute_performance will use try, and return NA if the function has an error. Of the potentially large number of cohort, score and bootstrap sample, this was a straightforward way to prevent compute_performance from returning an error for the whole dataset. However, if the function to compute performance generally does not work (for example, if the package pROC is required but not loaded), this behavior also prevents warning message for, say, unloaded R packages from being printed. See the example for how to redefine the provided functions with a transformation.

Value

A single performance measure (numeric).

Functions

• calibration_slope: Estimate calibration slope

• calibration_large: Estimate calibration-in-the-large

• c_statistic: Estimate c-Statistics / Area under the ROC curve

• oe_ratio: Estimate ratio of observed to expected number of events

• brier_score: Estimate Brier score

• int_calib_index: Estimate Integrated Calibration Index (ICI) (mean)

• E50: Estimate Integrated Calibration Index (ICI) (median)

• E90: Estimate Integrated Calibration Index (ICI) (90th percentile)

• Emax: Estimate Integrated Calibration Index (ICI) (maximum)

See Also

Austin, PC, Steyerberg, EW. The Integrated Calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models. Statistics in Medicine. 2019; 1– 15. https://doi.org/10.1002/sim.8281

Debray, T. P., Damen, J. A., Riley, R. D., Snell, K., Reitsma, J. B., Hooft, L., … Moons, K. G. (2018). A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes. Statistical Methods in Medical Research. https://doi.org/10.1177/0962280218785504

Examples

n <- 100
x <- rnorm(n)
y <- as.numeric(rnorm(n, x) > 1)
dat <- data.frame(x, y)

# log calibration slope
log_cs <- function(dd, fm){
    log(calibration_slope(dd, fm))
}

calibration_slope(dat, "y ~ x")
log_cs(dat, "y ~ x")

srhaile/mscpredmodel documentation built on Sept. 13, 2019, 3:44 p.m.