This package provides internal bootstrap validation for linear, logistic, cox and multinomial 'glmnet' models as well as lm and glm models.


vboot(glmnet_fit, x, y, s, nfolds = 5, B = 200, cv_replicates = 100, n_cores = max(1, parallel::detectCores() - 1))



Main objective of a predictive model is to provide accurated predictions of a new observations. Unfortunately we donĀ“t know how well the model performs. In addition, at the current era of omic data where p >> n is not reasonable applying internal validation using data-splitting. Under this background a good method to assessing model performance is applying internal bootstrap validation. The followed approach is described in Harrel et al. (1996) and on the fantastic blog written by Jonathan Bartlett. The bootstrap validation procedure consists of the following steps.

  1. Fit the model to original data, and estimate the measure of predictive accuracy A (for example AUC from the ROC curve in case of binary outcome or R^2 for numeric outcome). Denote this as A{orig}
  2. Repeat this process almost B = 100 or 200 times

    • Make a bootstrap sample from the original data
    • Fit the model to the bootstrap sample, and estimate A using the fitted model on the bootstrap sample. Denote this as A_b
    • Estimate A by applying the fitted bootstrap model on the original dataset. Denote this as A{b,orig}
  3. Calculate the estimate of optimism O = B^{-1} \sum_{b=1}^B A_b - A_{b,orig}

  4. Calculate the optimism adjusted as A_{orig} - O


The following example applies internal bootstrap validation on glmnet logistic regression.

On one hand, we create data.frame (x) storing the predictors, and y the binary outcome. model.matrix is required to glmnet function.

 x <- data.frame(matrix(rnorm(200), ncol = 200, nrow = 20), factor = factor(rep(c("A", "B"), each = 10)))
 y <- rep(c("Yes", "No"), each = 10)
 x <- model.matrix(~., data = x)
 On second hand, we fit a penalized elastic net logistic model with alpha = 0.5. 

 cv.lognet <- cv.glmnet(x, y, alpha = 0.5, nfolds = 5, family = "binomial")
 l <- cv.lognet$lambda.1se
 fit_lognet <- glmnet(x, y, alpha = 0.5, family = "binomial")
 Finally apply `vboot` function to get an original AUC, Optimism and adjusted AUC. 

 vboot(fit_lognet, x, y, nfolds = 3, B = 100, s=l, cv_replicates = 20, n_cores = 3)

Further lines

BootValidation package is still on development.

Ancamar/bootValidation documentation built on July 14, 2018, 10:55 a.m.