classification_stability: Classification instability plot

In pminternal: Internal Validation of Clinical Prediction Models

Classification instability plot

Description

Classification instability plot shows the relationship between original model estimated risk and the classification instability index (CII). The CII is the proportion of bootstrap replicates where the predicted class (0 if p <= threshold; 1 if p > threshold) is different to that obtained from the original model. Those with risk predictions around the threshold will exhibit elevated CII but an unstable model will exhibit high CII across a range of risk predictions. See Riley and Collins (2023).

Usage

classification_stability(
  x,
  threshold,
  xlim,
  ylim,
  xlab,
  ylab,
  pch,
  cex,
  col,
  subset,
  plot = TRUE
)

Arguments

x	an object produced by validate with method = "boot_*" (or boot_optimism with method="boot")
threshold	estimated risks above the threshold get a predicted 'class' of 1, otherwise 0.
xlim	x limits (default = range of estimated risks)
ylim	y limits (default = c(0, maximum CII))
xlab	a title for the x axis
ylab	a title for the y axis
pch	plotting character (default = 16)
cex	controls point size (default = 1)
col	color of points (default = grDevices::grey(.5, .5))
subset	vector of observations to include (row indices). This can be used to select a random subset of observations.
plot	if FALSE just returns CII values (see value)

Value

plots classification (in)stability. Invisibly returns estimates of CII for each observation.

References

Riley, R. D., & Collins, G. S. (2023). Stability of clinical prediction models developed using statistical or machine learning methods. Biometrical Journal, 65(8), 2200302. doi:10.1002/bimj.202200302

Examples

set.seed(456)
# simulate data with two predictors that interact
dat <- pmcalibration::sim_dat(N = 2000, a1 = -2, a3 = -.3)
mean(dat$y)
dat$LP <- NULL # remove linear predictor

# fit a (misspecified) logistic regression model
m1 <- glm(y ~ ., data=dat, family="binomial")

# internal validation of m1 via bootstrap optimism with 10 resamples
# B = 10 for example but should be >= 200 in practice
m1_iv <- validate(m1, method="boot_optimism", B=10)

classification_stability(m1_iv, threshold=.2)

pminternal documentation built on April 4, 2025, 3:30 a.m.