confint.internal_validatesummary: Confidence intervals for bias-corrected performance measures

In pminternal: Internal Validation of Clinical Prediction Models

Confidence intervals for bias-corrected performance measures

Description

Implements the methods discussed in Noma et al. (2021), plus some others that have not been tested. Specifically, Noma et al. discuss bootstrap optimism correction ("boot_optimism" and ".632") and the percentile bootstrap (ci_type = "perc"). Their paper contains some simulation results on coverage properties of these CIs. If you used validate to do something other than bootstrap optimism correction or if you request normal approximation CIs please note that these approaches have (to my knowledge) not been thoroughly tested. ci_type = "norm" is included as it might be able to reduce the number of runs needed for "twostage" CIs. See details for the difference between "shifted" and "twostage". "norm" CIs are likely to perform poorly for some performance measures, such as calibration Intercept and Slope, which for regular glms are always 0 and 1, respectively, on assessment of apparent performance. As "shifted" CIs are based on apparent performance they will be meaningless for these measures. Use the untested methods with caution!

Usage

## S3 method for class 'internal_validate'
confint(
  object,
  parm,
  level = 0.95,
  method = c("shifted", "twostage"),
  ci_type = c("perc", "norm"),
  R = 1000,
  add = TRUE,
  ...
)

Arguments

object	created by call to validate
parm	a specification of which performance measures are to be given confidence intervals, either a vector of numbers or a vector of names. If missing, all scores are considered.
level	the confidence level required
method	"shifted" or "twostage" (see details)
ci_type	percentile ("perc") or normal approximation ("norm") bootstrap CIs
R	number of replicates
add	return the object with an additional slot containing CIs (default) or just return the CIs
...	additional arguments (currently ignored)

Details

The two methods are as follows (see Noma et al. (2021) for more details):

shifted

(default) This approach is based on shifting bootstrap CIs for apparent performance by optimism. This makes it the faster option as only the calculation of apparent performance is needed for each replicate. If the CI for apparent performance is [lower, upper], the resulting CI for bias-corrected performance is [lower - optimism, upper - optimism]. Note this method is only available when using an optimism based approach (and "cv_optimism" was untested in Noma et al).

twostage

This approach creates a bootstrap resample of the data and runs the entire validation procedure on the resample (with the same number of 'inner' replicates, determined by B in the original validate call). The CI is then constructed using the corrected estimates from the R 'outer' replicates. As this involves R*B replicates, this could take a long time. Note validate takes a cores argument that can allow the inner samples to run in parallel.

Value

A list with two elements, each a matrix with columns giving lower and upper confidence limits for each measure. One for apparent and one for bias-corrected measures. Columns will be labelled as (1-level)/2 and 1 - (1-level)/2 in % (by default 2.5% and 97.5%).

References

Noma, H., Shinozaki, T., Iba, K., Teramukai, S., & Furukawa, T. A. (2021). Confidence intervals of prediction accuracy measures for multivariable prediction models based on the bootstrap‐based optimism correction methods. Statistics in Medicine, 40(26), 5691-5701.

pminternal documentation built on April 4, 2025, 3:30 a.m.