This package provides internal bootstrap validation for lineal, logistic, cox and multinomial 'glmnet' models, as well as lm and glm (binomial) regression.
vboot(glmnet_fit, x, y, s, nfolds = 5, B = 200, cv_replicates = 100, n_cores = max(1, parallel::detectCores() - 1))
glmnet_fit
: Object from glmnet fit x
: A matrix of the predictors, each row is an observation vector.y
A vector of response variable.s
Value of the penalty parameter "lambda" selected from the original cv.glmnetnfolds
Number of folds for cross validation as in cv.glmnetB
Number of bootsrap samplescv_replicates
Number of replicates for the cross-validation stepn_cores
number of cores to use in parallel. Default detectCores()-1Main objective of a predictive model is to provide accurated predictions of a new observations. Unfortunately we donĀ“t know how well the model performs. In addition, at the current era of omic data where p >> n is not reasonable applying internal validation using data-splitting. Under this background a good method to assessing model performance is applying internal bootstrap validation. The followed approach is described in Harrel et al. (1996) and on the fantastic blog written by Jonathan Bartlett. The bootstrap validation procedure consists of the following steps.
Repeat this process almost B = 100 or 200 times
Calculate the estimate of optimism O = B^{-1} \sum_{b=1}^B A_b - A_{b,orig}
The following example applies internal bootstrap validation on glmnet
logistic regression.
On one hand, we create data.frame
(x
) storing the predictors, and y
the binary outcome. model.matrix
is required to glmnet
function.
x <- data.frame(matrix(rnorm(200), ncol = 200, nrow = 20), factor = factor(rep(c("A", "B"), each = 10)))
y <- rep(c("Yes", "No"), each = 10)
x <- model.matrix(~., data = x)
```
On second hand, we fit a penalized elastic net logistic model with alpha = 0.5.
```{r}
library(glmnet)
cv.lognet <- cv.glmnet(x, y, alpha = 0.5, nfolds = 5, family = "binomial")
l <- cv.lognet$lambda.1se
fit_lognet <- glmnet(x, y, alpha = 0.5, family = "binomial")
```
Finally apply `vboot` function to get an original AUC, Optimism and adjusted AUC.
```{r}
vboot(fit_lognet, x, y, nfolds = 3, B = 100, s=l, cv_replicates = 20, n_cores = 3)
BootValidation
package is still on development. The next aim is to set new features to vboot function like internal validation for non penalized regressions.
Any scripts or data that you put into this service are public.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.