calplot: Construct calibration plots for a nested.glmnetr output...

In glmnetr: Nested Cross Validation for the Relaxed Lasso and Other Machine Learning Models

Construct calibration plots for a nested.glmnetr output object

Description

Using k-fold cross validation this function constructs calibration plots for a nested.glmnetr output object. Each hold out subset of the k-fold cross validation is regressed on the x*beta predicteds based upon the model fit using the non-hold out data using splines. This yields k spline functions for evaluating model performance. These k spline functions are averaged to provide an overall model calibration. Standard deviations of the k spline fits are also calculated as a function of the predicted X*beta, and these are used to derive and plot approximate 95 (mean +/- 2 * SD/sqrt(k)). Note, standard errors calculated in this manner may underestimate (or overestimate?) the true standard error, the displayed confidence intervals might be too narrow for a 95 probability and should be interpreted with caution. See the package vignettes for discussion and references. Further, because regression equations can be unreliable when extrapolating beyond the data range used in model derivation, we display this overall calibration fit and CIs with solid lines only for the region which lies within the ranges of the predicted x*betas for all the k leave out sets. The spline fits are made using the same framework as in the original machine learning model fits, i.e. one of "cox", "binomial" or "gaussian"family. For the "cox" framework the pspline() funciton is used, and for the "binomial" and "gaussian" frameworks the ns() function is used. Predicted X*betas beyond the range of any of the hold out sets are displayed by dashed lines to reflect the lessor certainty when extrapolating even for a single hold out set.

Usage

calplot(
  object,
  wbeta = NULL,
  df = 3,
  resample = NULL,
  oob = 1,
  bootci = 0,
  plot = 1,
  plotfold = 0,
  plot_full = 0,
  plothr = 0,
  knottype = 1,
  trim = 0,
  vref = 0,
  xlim = NULL,
  ylim = NULL,
  xlab = NULL,
  ylab = NULL,
  col.term = 1,
  col.se = 2,
  rug = 1,
  seed = NULL,
  cv = NULL,
  fold = NULL,
  ...
)

Arguments

object	A nested.glmnetr() output object for calibration
wbeta	Which Beta should be plotted, an integer. This will depend on which machine learning models were run when creating the output object. If unsure the user can run the function without specifying wbeta and a legend will be directed to the console.
df	The degrees of freedom for the spline function
resample	1 to base the splines on the leave out X*Beta's ($xbetas.cv or $xbetas.boot.oob), or 0 to use the naive X*Beta's ($xbetas). This can be done to see biases associated with the naive approach.
oob	1 (default) to construct calibration plots using the out-of-bag data points, 0 to use in bag (including resampled data points) data points. This option only applies when bootstrap is used instead of k-fold cross validation, and when resample is set to 1. For cross validation evaluations out-of-bag samples (folds) are always used for evaluation. The purpose of oob = 0 is to allow evaluation of the variability of bootstrap calibrations ignoring bias like done in Riley et al., 2023, doi: 10.1186/s12916-023-03212-y and Austin and Steyerberg 2013, doi: 10.1002/sim.5941
bootci	1 to calculate bootstrap confidence intervals for calibration curves adjusting for bias, 0 (default) to simply plot the calibration curves based upon the inbag data. This is for exploration only, and only when bootstrap samples were used for model performance evaluation. The applicability of bootstrap confidence intervals for these calibration curves is questionable. If bootci is set to 1 then oob is set to 0.
plot	1 by default to produce plots, 0 to output data for plots only, 2 to plot and output data.
plotfold	0 by default to not plot the individual fold calibrations, 1 to overlay the k leave out spline calibration fits in a single figure and 2 to produce separate plots for each of the k hold out calibration curves.
plot_full	plot full data
plothr	a power > 1 determining the spacing of the values on the axes, e.g. 2, exp(1), sqrt(10) or 10. The default of 0 plots the X*Beta. This only applies fore "cox" survival data models.
knottype	1 (default) to use XBeta used for the spline fit to choose knots in ns() for gaussian and binomial families, 2 to use the XBeta from all re-samples to determine the knots.
trim	the percent of top and bottom of the data to be trimmed away when producing plots. The original data are still used used calcualting the curves for plotting.
vref	Similar to trim but instead of trimming the spline lines, plots vertical refence lines aht the top vref and bottom vref percent of the model X*Betas's
xlim	xlim for the plots. This does not effect the curves within the plotted region. Caution, for the "cox" framework the xlim are specified in terms of the X*beta and not the HR, even when HR is described on the axes.
ylim	ylim for the plots, which will usually only be specified in a second run of for the same data. This does not effect the curves within the plotted region. Caution, for the "cox" framework the ylim are specified in terms of the X*beta and not the HR, even when HR is described on the axes.
xlab	a user specified label for the x axis
ylab	a user specified label for the y axis
col.term	a number for the line depicting the overall calibration estimates
col.se	a number for the line depicting the +/- 2 * standard error lines for the overall calibration estimates
rug	1 to plot a rug for the model x*betas, 0 (default) to not.
seed	an integer seed used to random select the multiple of X*Betas to be used in the rug when using bootstraping for model evaluation as sample elements may be included multiple times as test (Out Of Bag) data.
cv	Deprecated. Use resample option instead.
fold	Deprecated. This term is now ignored.
...	allowance to pass terms to the invoked plot function

Details

Optionally, for comparison, the program can fit a spline based upon the predicted x*betas ignoring the cross validation structure, or one can fit a spline using the x*betas calculated using the model based upon all data.

Value

Calibration plots are returned by default, and optionally data for plots are output to a list.

Author(s)

Walter Kremers ([e-mail])

See Also

plot.nested.glmnetr , summary.nested.glmnetr , nested.glmnetr

glmnetr documentation built on April 3, 2025, 6:45 p.m.