Nothing
#' Weighted XGBoost prediction models (wXGBoost) for complex survey data
#'
#'@description A model selection function for fitting XGBoost (linear or logistic) models for complex survey data with sampling weights
#' by a replicating weights method to select optimal tuning parameters by weighted error minimization.
#' Detailed arguments are referred to in [xgboost::xgboost()].
#'
#' @param data A data frame with information about the response variable and covariates, as well as sampling weights, strata, and cluster indicators. It could be \code{NULL} if the sampling design were plugged into the \code{design} argument.
#' @param y A vector of response variable.
#' @param col.x A numeric vector indicating indices of the covariates or a string vector indicating these column names.
#' @param cluster A string of a cluster identifier. It could be \code{NULL} if the sampling design were plugged in the \code{design} argument.
#' @param strata A string of a strata identifiers. It could be \code{NULL} if the sampling design were plugged in the \code{design} argument.
#' @param weights A string of a sampling weights' identifier. It could be \code{NULL} if the sampling design were plugged in the \code{design} argument.
#' @param design An object of class \code{survey.design} generated by \code{survey::svydesign()}. It could be \code{NULL} if information about \code{cluster}, \code{strata}, \code{weights} and \code{data} were given.
#' @param params The list of parameters, or the list obtained by [xgboost::xgb.params()]. Should include \code{objective} to be a string of either `binary:logistic` for binary classification or `reg:squarederror` for linear regression.
#' Default evaluation metrics assigned by \code{params:objective} are weighted mse ([evalerror_reg()]) for regression,
#' weighted mean error ([evalerror_bin()]) for logistic reg (binary classification). The complete list of parameters is found on
#' [xgboost::xgb.train()] or \url{https://xgboost.readthedocs.io/en/latest/parameter.html}.
#' @param maximize If `early_stopping_rounds` is set, then this parameter must be set as well. Default is `FALSE` since the weighted loss function is used for minimization.
#' This parameter is passed to the \code{xgboost::cb.early.stop} callback.
#' @param nrounds The maximum number of boosting iterations.
#' @param print_every_n Print each n-th iteration evaluation messages when verbose>0. Default is `1` printing all messages. This parameter is passed to the `cb.print.evaluation` callback.
#' @param verbose `0` for xgboost to stay silent. Default= `1` for printing information about performance. `2` for printing some additional information out. Note that setting verbose > 0 automatically engages the `cb.print.evaluation`(period=1) callback function.
#' @param early_stopping_rounds \code{NULL} does not trigger the early stopping function. Setting to an integer k will stop training with a validation set unless the performance improve for k rounds. Setting this parameter engages the `cb.early.stop` callback.
#' @param missing Default is set to `NA`, which means that `NA` values should be considered as 'missing' by the algorithm. Sometimes, `0` or other extreme values might be used to represent missing values. This parameter is only used when input is a dense matrix.
#' @param final.model Boolean. Default is \code{TRUE} printing out the object of class \code{xgb.Booster} for the final model selected by replicate weights' CV with corresponding predicted values.
#' @param test.data Optional. An object of `xgb.DMatrix`. It can be used to fit the final model with the best iteration for evaluation and to estimate predicted values only if `final.model = TRUE`.
#' @param method A character string indicating a method of replicate weights. Choose one of these: \code{JKn}, \code{dCV}, \code{bootstrap}, \code{subbootstrap}, \code{BRR}, \code{split}, \code{extrapolation}.
#' @param k An integer. The number of folds for the \code{dCV} method. Default is \code{k=10}.
#' @param R An integer. The number of times the sample is partitioned for \code{dCV}, \code{split} or \code{extrapolation} method. Default is \code{R=1}.
#' @param B An integer. The number of bootstrap re-samples for \code{bootstrap} and \code{subbootstrap} methods. Default is \code{B=200}.
#' @param dCV.sw.test A logical value indicating the method for estimating the error for \code{dCV} method. \code{FALSE}, (the default option) estimates the error for each test set and defines the cross-validated error based on the average strategy.
#' \code{TRUE} estimates the cross-validated error based on the pooling strategy.
#' @param train.prob A numeric between 0 and 1, indicating the proportion of clusters (for the method \code{split}) or strata (for the method \code{extrapolation}) to be set in the training sets. Default is \code{train.prob = 0.7}. Only applies to \code{split} and \code{extrapolation} methods.
#' @param method.split A string of one the following replicate weights methods to be implemented under the \code{split} in the \code{method} argument: \code{dCV}, \code{bootstrap} or \code{subbootstrap}.
#' @param print.rw A logical value. If \code{TRUE}, the data set with the replicate weights is saved in the output object. Default \code{print.rw=FALSE}.
#' @param save_name The name or path for periodically saved model file.
#' @param wxgb_model A previously built model to continue the training from. Could be either an object of class \code{xgb.Booster}, i.e., \code{final.model} from the object of class \code{w.xgboost}, its raw data, or the name of a file with a previously saved model.
#' @param save_period When it is non-NULL, model is saved to disk after every `save_period` rounds, `0` means save at the end. The saving is handled by the `cb.save.model` callback.
#' @param callbacks A list of callback functions to perform various task during boosting. See \code{xgboost::callbacks}. Some of the callbacks are automatically created depending on the parameters’ values.
#' Users can provide either existing or their own callback methods in order to customize the training process.
#'
#'
#' @seealso [xgboost::xgb.train()] for relevant arguments and return values in detail.
#' @seealso [eval_loss()] for additional customized function for weighted logloss.
#'
#'
#' @return The output object of the function \code{wXGBoost()} is an object of class \code{w.xgboost}:
#' - `CV.iterations`: A list containing information on tuning parameters, `nrounds` of three elements:
#' - `niter.range.per.R`: A matrix of the range of iterations of replicate-weight CV for `R` replicates.
#' - `fold.best`: A list of length `R`; each element \code{fold.best[[r]]} is a numeric vector of best iterations (one per fold/replicate-weight column actually produced for replicate `r`, minimizing the average error). The length of this vector depends on the selected \code{method} and is **not** fixed at \code{k}: it equals \code{k} itself only for \code{method = "dCV"}; for \code{"JKn"} it equals the number of PSUs; for \code{"bootstrap"}/\code{"subbootstrap"} it equals \code{B}; for \code{"BRR"} it is a power-of-4 count determined internally by \code{survey::as.svrepdesign()}; and for \code{"split"}/\code{"extrapolation"} it is \code{1}. (Named \code{fold.best}, not \code{k_fold.best}, precisely because it is not always a "k-fold" count in this general sense -- \code{k} itself only has meaning for \code{dCV}.)
#' - `best_iteration`: An optimal `nround` that minimizes mean test error.
#' - `CV.eval_log`: A list containing information of two elements:
#' - `CV`: A matrix of iteration-wise mean and sd for training and test sets, respectively.
#' - `Best_iteration`: A vector of minimum \code{CV.eval_log$CV} to pick up the final model corresponding to \code{CV.iteratons$best_iteration}.
#' - `final.model`: A list containing information on the fitted model by \code{Best_iteration}: an object of class \code{xgb.Booster}.
#' Note that the final model is fitted by the original \code{data}, not one of the generated training sets by replicate weights.
#' When `test.data` is provided, the final model is fitted by evaluating model performance.
#' - `predicted`: A vector of predicted values with training set, `data` (`test.data = NULL`) or a list of respective predicted values with training set, `data` and test set, `test.data` from `final.model= TRUE`.
#' - `data.rw`: A data frame containing the original data set and the replicate weights added to define training and test sets. Only included in the output object if \code{print.rw=TRUE}.
#' - `call`:The call that executes this function producing the object of \code{w.xgboost}.
#'
#' @note
#' `final.model` can leverage R-\code{xgboost} functionality for comprehensive data analysis for linear/logistic regression but will be
#' extensively used for other models, i.e., Cox, Poisson, etc.
#'
#' @note
#' \code{method = "bootstrap"} and \code{"subbootstrap"} rely on
#' \code{\link{replicate_weights}}'s out-of-bag (OOB) style test weights
#' (\code{sw_r_1_test_*} columns): for each of the \code{B} replicates, a
#' unit is treated as "test" for that replicate if and only if its PSU was
#' not drawn into that resample. The OOB test fraction per replicate is
#' therefore random rather than a fixed, predetermined proportion (unlike
#' \code{dCV} or \code{split}).
#'
#' @note
#' \strong{Breaking change (if upgrading from an earlier version):} the
#' \code{result$CV.iterations} field previously named \code{k_fold.best} is
#' now named \code{fold.best}, and its contents changed from a pre-sized
#' \code{k x R} matrix to a list of length \code{R} (with per-replicate
#' vectors whose length depends on \code{method}, not fixed at \code{k}).
#' This was a necessary bug fix: the old matrix, pre-sized assuming exactly
#' \code{k} folds regardless of \code{method}, either silently corrupted
#' results (for \code{split}/\code{extrapolation}, where the real fold count
#' is 1, leaving most matrix rows as fake zeros) or crashed with "subscript
#' out of bounds" (for \code{JKn}/\code{BRR}/\code{bootstrap}/
#' \code{subbootstrap}, where the real fold count is almost always greater
#' than the default \code{k=10}). Code that previously read
#' \code{fit$CV.iterations$k_fold.best} as a matrix must be updated to read
#' \code{fit$CV.iterations$fold.best} as a list, e.g.
#' \code{fit$CV.iterations$fold.best[[r]]} for replicate \code{r}'s vector
#' of best iterations.
#'
#' @note
#' \strong{On the argument \code{k}:} \code{k} denotes the number of folds
#' \emph{only} for \code{method = "dCV"} (its only meaningful use, matching
#' \code{\link{replicate_weights}}'s own documentation of \code{k}). For every
#' other \code{method}, \code{k} is ignored entirely (a message is printed if
#' a non-default value is supplied), and the real number of replicate-weight
#' columns used internally is determined by the method itself as described
#' under \code{fold.best} above.
#'
#' @examples
#'
#' # Global options to avoid only one PSU in a stratum in a particular domain or subpopulation
#' options(survey.adjust.domain.lonely=TRUE)
#' options(survey.lonely.psu="adjust")
#'
#'data(nhanes2013_sbc)
#'
#' # Selecting optimal (hyper-) parameters for a model development:
#' param <- list(objective = "binary:logistic", max_depth = sample(4:8, 1),
#' seed = sample.int(10000,1),
#' eta = runif(1, .01, .3),gamma = runif(1, 0.0, 1),
#' subsample = 0.5, # much smaller for large N
#' colsample_bytree = runif(1, .5, .8),
#' min_child_weight = sample(1:40, 1),
#' max_delta_step = sample(1:10, 1)) # for very imbalanced case
#' # Or equivalently,
#' param<- xgboost::xgb.params(objective = "binary:logistic", max_depth = sample(4:8, 1),
#' subsample = 0.5)
#'
#' # Variable selection and best iteration (hyper-parameter) selection:
#' # Can add a xgb.DMatrix object of test data (`test.data`) to the function
#' # to obtain predicted values from the final model after extra evaluation with it.
#' \donttest{
#' xgb.cv <- wXGBoost(data = nhanes2013_sbc,
#' y = nhanes2013_sbc$HBP, col.x = 2:61,
#' cluster = "SDMVPSU", strata = "SDMVSTRA", weights = "WTSAF2YR",
#' params = param, nrounds =10000,verbose=0,early_stopping_rounds=5,
#' method = "dCV", k=10, R=20)
#' }
#'\donttest{
#' # Or equivalently:
#' des <- survey::svydesign(ids=~SDMVPSU, strata = ~SDMVSTRA, weights = ~WTSAF2YR,
#' nest = TRUE, data = nhanes2013_sbc)
#' xgb.cv <- wXGBoost(y = nhanes2013_sbc$HBP, col.x = 2:61, design = des,
#' params = param, nrounds =10000,verbose=0,early_stopping_rounds=5,
#' method = "dCV", k=10, R=20)
#' }
#'@export
wXGBoost <- function(data = NULL, y =NULL, col.x = NULL,missing = NA,final.model=TRUE,test.data=NULL,
cluster = NULL, strata = NULL, weights = NULL, design = NULL,
params = list(), nrounds, verbose = 0, print_every_n = 1L,
maximize = FALSE, early_stopping_rounds = 8,
method = c("dCV", "JKn", "bootstrap", "subbootstrap", "BRR", "split", "extrapolation"),
k = 10, R = 1, B = 200, dCV.sw.test = FALSE,
train.prob = 0.7, method.split = c("dCV", "bootstrap", "subbootstrap"),
print.rw = FALSE, save_period = NULL, save_name = "wxgboost.model",
wxgb_model = NULL, callbacks = list()){
# Stops and messages:
if(is.null(data) & is.null(design)){stop("Information about either the data set ('data') or the sampling design ('design') needed.")}
if(method == "split"){
if(is.null(train.prob)){stop("Selected replicate weights method: 'split'.\nPlease, set a value between 0 and 1 for the argument 'train.prob'.")}
if(train.prob < 0 | train.prob > 1){stop("Selected replicate weights method: 'split'.\nPlease, set a value between 0 and 1 for the argument 'train.prob'.")}
if(length(method.split)!=1){stop("Selected replicate weights method: 'split'.\nPlease, set a valid method for the argument 'method.split'. Choose between: 'dCV', 'bootstrap' or 'subbootstrap'.")}
}
if(method == "extrapolation"){
if(is.null(train.prob)){stop("Selected replicate weights method: 'extrapolation'.\nPlease, set a value between 0 and 1 for the argument 'train.prob'.")}
if(train.prob < 0 | train.prob > 1){stop("Selected replicate weights method: 'extrapolation'.\nPlease, set a value between 0 and 1 for the argument 'train.prob'.")}
}
if(method %in% c("JKn", "bootstrap", "subbootstrap", "BRR")){
if(R!=1){message("Selected method: ", method,". For this method, R = 1. Thus, the argument R = ",R, " has been ignored.")}
}
if(method %in% c("dCV", "split", "extrapolation")){
if(R != round(R)){stop("The argument 'R' must be an integer greater or equal to 1. R=",R," is not an integer.\nPlease, set a valid value for 'R' or skip the argument to select the default option R=1.")}
if(R < 1){stop("The argument 'R' must be an integer greater or equal to 1. R=",R," lower than 1.\nPlease, set a valid value for 'R' or skip the argument to select the default option R=1.")}
}
if(method != "dCV"){
if(!is.null(k) & k!=10){message("Selected method: ", method,". The argument k = ",k, " is not needed and, hence, has been ignored.")}
}
if(method == "dCV"){
if(k != round(k)){stop("The argument 'k' must be an integer. k=",k," is not an integer.\nPlease, set a valid value for 'k' or skip the argument to select the default option k=10.")}
if(k < 1){stop("The argument 'k' must be a positive integer. k=",k," is not a positive integer.\nPlease, set a valid value for 'k' or skip the argument to select the default option k=10.")}
}
if(!(method %in% c("bootstrap", "subbootstrap"))){
if(!is.null(B) & B!=200){message("Selected method: ", method,". The argument B = ",B, " is not needed and, hence, has been ignored.")}
}
if(method %in% c("bootstrap", "subbootstrap")){
if(B != round(B)){stop("The argument 'B' must be an integer. B=",B," is not an integer.\nPlease, set a valid value for 'B' or skip the argument to select the default option B=200.")}
if(B < 1){stop("The argument 'B' must be a positive integer. B=",B," is not a positive integer.\nPlease, set a valid value for 'B' or skip the argument to select the default option B=200.")}
}
# Step 0: Notation
if(!is.null(design)){
cluster <- as.character(design$call$id[2])
if(cluster == "1" || cluster == "0"){
cluster <- NULL
}
strata <- as.character(design$call$strata[2])
weights <- as.character(design$call$weights[2])
data <- design$variables
}
# Step 1: Generate replicate weights based on the method
newdata <- replicate_weights(data = data, method = method,
cluster = cluster, strata = strata, weights = weights,
k = k, R = R, B = B,
train.prob = train.prob, method.split = method.split,
rw.test = TRUE, dCV.sw.test = dCV.sw.test)
# Step 2: Do CV to find the best iteration
mat<- as.matrix(data[,col.x])
xgb_model <- NULL
Niter<-NULL; Niter.max<-NULL
# fold.best.iter is a list (length R), not a pre-sized matrix: the
# number of folds/replicate-weight columns actually produced per
# replicate 'r' depends on the selected method (k for dCV, number of
# PSUs for JKn, B for bootstrap/subbootstrap, a power-of-4 count for
# BRR, 1 for split/extrapolation) and is not necessarily equal to the
# 'k' argument, which is specific to dCV's fold count. A matrix
# pre-sized with nrow = k would silently corrupt output (when the real
# fold count < k) or error with "subscript out of bounds" (when the
# real fold count > k, e.g. almost always true for JKn/BRR/bootstrap).
# Named fold.best (not k_fold.best) since it does not represent a
# "k-fold" count in this general sense -- 'k' itself is only meaningful
# for method = "dCV".
fold.best.iter <- list()
k.fold.error <- list();
#attach eval.nonpara.R for testing measure
# param custom_metric User defined evaluation function. Returns list(metric='metric-name', value='metric-value') with given prediction and dtrain.
# Default evaluation metric is assigned by \code{params:objective}: weighted mse ([evalerror_reg()]) for regression,
# weighted mean error ([evalerror_bin()]) for logistic reg or binary classification.
# if(is.null(custom_metric)){
if(params$objective == "binary:logistic"){
eval.metric<- evalerror_bin
# eval.metric<- eval.loss
}else {
eval.metric<- evalerror_reg
}
# }else {
# eval.metric<-custom_metric
# }
for (r in 1:R){
rwtraincols <- grep(paste0("r_",r,"_train"), colnames(newdata))
rwtestcols <- grep(paste0("r_",r,"_test"),names(newdata))
eval.log<-list();total.iter <- NULL;
# Actual fold/replicate count for this r, used throughout instead of
# the dCV-specific 'k' argument.
n.folds <- length(rwtraincols)
fold.best.iter[[r]] <- numeric(n.folds)
for(k.w in 1:n.folds){
#generate supporting type data
dtrain<- xgb.DMatrix(data = mat, label = y, weight = newdata[,rwtraincols[k.w]], nthread = params$nthread, missing = missing)
dtest <- xgb.DMatrix(data = mat, label = y, weight = newdata[,rwtestcols[k.w]], nthread = params$nthread, missing = missing)
watchlist <- list(train=dtrain, eval=dtest)
model <- xgboost::xgb.train(params = params, data = dtrain, evals = watchlist,
nrounds=nrounds, custom_metric = eval.metric,
verbose = verbose, print_every_n = print_every_n,
early_stopping_rounds = early_stopping_rounds, maximize = maximize,
save_period = save_period, save_name = save_name,
xgb_model = wxgb_model, callbacks = list())
eval.log[[length(eval.log)+1]]<-attributes(model)$evaluation_log
names(eval.log)[[length(eval.log)]]<- names(newdata)[rwtraincols[k.w]]
# if(!is.null(early_stopping_rounds)){
fold.best.iter[[r]][k.w] <- attributes(model)$early_stop$best_iteration
total.iter <- c(total.iter,dim(attributes(model)$evaluation_log)[1])
# } else{ total.iter <- model$niter
# }
}
#find min error per r to pick best nrounds from the replicate-weight CV
min.niter<- min(total.iter)
train.eval<- NULL
test.eval <- NULL
# Use the actual number of folds produced for this replicate
# (length(eval.log), == n.folds above), not the 'k' argument -- see
# note on k.fold.best.iter above for why this must not be hardcoded.
for (j in 1:n.folds){
train.eval <- cbind(train.eval,eval.log[[j]][1:min.niter,2])
test.eval <- cbind(test.eval,eval.log[[j]][1:min.niter,3])
}
#collect min niters and corresponding iteration-wise mean for k-folds per r
Niter<- c(Niter, min.niter); Niter.max<-c(Niter.max,max(total.iter))
k.fold.error[[r]] <- cbind(apply(train.eval,1,mean,na.rm=TRUE),apply(test.eval,1,mean,na.rm=TRUE))
}
min_nstops <- min(Niter)
train.mean.error<-NULL;test.mean.error<-NULL
for(r in 1:R){
train.mean.error<- cbind(train.mean.error,k.fold.error[[r]][1:min_nstops,1])
test.mean.error<- cbind(test.mean.error,k.fold.error[[r]][1:min_nstops,2])
}
mean.train.error<- apply(train.mean.error,1,mean,na.rm=TRUE)
sd.train.error <- apply(train.mean.error,1,sd,na.rm=TRUE)
mean.test.error <- apply(test.mean.error,1,mean,na.rm=TRUE)
sd.test.error <- apply(test.mean.error,1,sd,na.rm=TRUE)
Eval.log <- cbind(mean.train.error,sd.train.error,mean.test.error,sd.test.error)
Eval.log<- data.frame(itr=1:min_nstops,Eval.log)
best_iter <- which.min(mean.test.error)
#step 3: run the final model
if(final.model){
#generate supporting data type
dtrain<- xgb.DMatrix(data = mat, label = y, weight = data[,weights],nthread = params$nthread, missing=missing )
# Always include at least 'train' in eval.list. This ensures 'evals' is
# never empty -- required by xgboost whenever early_stopping_rounds is
# non-NULL. We also disable early stopping for this final fit below,
# since nrounds = best_iter was already determined to be optimal by the
# replicate-weights CV above; there is nothing left to "stop early" on,
# and re-enabling it here could only truncate training before best_iter.
if(is.null(test.data)){
eval.list<- list(train = dtrain)
}else{ #!is.null(test.data) case:
eval.list <-list(train=dtrain,test=test.data)
}
model <- xgboost::xgb.train(params = params, data = dtrain, evals = eval.list,
nrounds=best_iter, custom_metric = eval.metric,
verbose = verbose, print_every_n = print_every_n,
maximize = maximize,early_stopping_rounds = NULL,
save_period = save_period, save_name = save_name,
callbacks = list())
if(is.null(test.data)){
yhat <- predict(model, newdata=dtrain)
}else{
yhat<-list(train = predict(model, newdata=dtrain),
test = predict(model, newdata=test.data) )
}
}
result <- list()
result$CV.iterations <- list(niter.range.per.R = rbind(Niter,Niter.max),
fold.best=fold.best.iter,
best_iteration = best_iter)
result$CV.eval_log <- list(CV = Eval.log,
Best_iteration = Eval.log[best_iter,])
if(final.model) {result$final.model <- model
result$predicted <- yhat}
result$call <- match.call()
if(print.rw == TRUE){result$data.rw <- newdata}
class(result) <- "w.xgboost"
return(result)
}
Any scripts or data that you put into this service are public.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.