CI.raplot: Statistical metrics with confidence intervals

In JohnPickering/risk-assessment-plot-package: Risk Assessment Plot and reclassification metrics

Statistical metrics with confidence intervals

Description

The CI.raplot function produces summary metrics for risk assessment. Outputs the NRI, IDI, weighted NRI and category Free NRI all for those with events and those without events. Also the AUCs of the two models and the comparison (DeLong) between AUCs. Output includes confidence intervals. Uses statistics.raplot. Displayed graphically by raplot.

Usage

CI.raplot(
  x1,
  x2 = NULL,
  y = NULL,
  t = NULL,
  NRI_return = FALSE,
  conf.level = 0.95,
  n.boot = 1000,
  dp = 3
)

Arguments

x1	Either a logistic regression fitted using glm (base package) or lrm (rms package) or calculated probabilities (eg through a logistic regression model) of the baseline model. Must be between 0 & 1
x2	Either a logistic regression fitted using glm (base package) or lrm (rms package) or calculated probabilities (eg through a logistic regression model) of the new (alternative) model. Must be between 0 & 1
y	Binary of outcome of interest. Must be 0 or 1 (if fitted models are provided this is extracted from the fit which for an rms fit must have x = TRUE, y = TRUE).
t	The risk threshold(s) for groups. eg t<-c(0,0.1,1) is a two group model with a threshold of 0.1 & t<-c(0,0.1,0.3,1) is a three group model with thresholds at 0.1 and 0.3.
NRI_return	If NRI statistics are required (default = FALSE).
conf.level	The confidence interval expressed as a fraction of 1 (ie 0.95 is the 95% confidence interval )
n.boot	The number of "bootstraps" to use. Performance slows down with more bootstraps. For trialling result, use a low number (eg 5), for accuracy use a large number (eg 2000)
dp	The number of decimal places to display

Value

A list with four items:

• 1. meta_data Some overall meta data - Confidence Interval, number of bootstraps, thresholds, input type

• 2. Metrics Point estimates of the statistical metrics (see list below)

• 3. Each_bootstrap_metrics Point estimates of the statistical metrics for each bootstrapped sample (see list below)

• 4. Summary Metrics Point estimates with confidence intervals of the statistical metrics. See following list:)

  • Total (n) Total number of subjects

  • Events (n) Number of subjects with the event (outcome) of intrest

  • Non-events (n) Number of subjects without the event (outcome) of intrest

  • NRI events The NRI with confidence interval for those with the event.

  • NRI non-events The NRI with confidence interval for those without the event.

  • IDI events The IDI (Integrated Discrimination Improvement) with confidence interval for those with the event. Expressed as a fraction

  • IDI non-events The IDI with confidence interval for those without the event. Expressed as a fraction

  • IS(baseline model) The Integrated Sensitivity (area under the sensitivity-calculated risk curve) for the baseline model

  • IS(new model) The Integrated Sensitivity for the reference (alt) model. Note, the IDI events should be the difference between IS(new model) and IS(baseline model)

  • IP(baseline model) The Integrated 1-Specificity (area under the 1-specificity-calculated risk curve) for the baseline model

  • IP(new model) The Integrated S1-Specificity for the reference (alt) model. Note, the IDI non-events should be the difference between IP(new model) and IP(baseline model)

  • AUC(baseline model) The Area Under the Receiver Operator Characteristic Curve for the baseline model

  • AUC(new model) The Area Under the Receiver Operator Characteristic Curve for the new (alt) model

  • AUC difference The difference in the AUCs betwen the reference and new model with a confidence interval

  • difference (p) P value for the difference in AUCs (DeLong method)

  • Brier(baseline model) The Brier score for the baseline model

  • Brier(new model) The Brier score for the alternate model

  • Brier skill The percent improvement of the alternatve over the baseline model based on the relative change in Brier score

  • incidence The incidence of the event

References

Pencina, M. J., D'Agostino, R. B., & Vasan, R. S. (2008). Evaluating the added stats::predictive ability of a new marker: From area under the ROC curve to reclassification and beyond. Statistics in Medicine, 27(2), 157–172. doi:10.1002/sim.2929

Examples

## Not run: 
data(data_risk)
y<-data_risk$outcome 
x1<-data_risk$baseline
x2<-data_risk$new
t<-c(0,0.19,1) 
#e.g.
output<-CI.raplot(x1, x2, y, t, conf.level = 0.95, n.boot = 5, dp = 2) 

## End(Not run)

JohnPickering/risk-assessment-plot-package documentation built on July 3, 2023, 8:41 a.m.