l2boost-package: Efficient implementation of Friedman's boosting algorithm for...

In ehrlinger/l2boost: Exploring Friedman's Boosting Algorithm for Regularized Linear Regression

Efficient implementation of Friedman's boosting algorithm for linear regression using an l2-loss function and coordinate direction (design matrix columns) basis functions.

Description

The l2boost package implements a generic boosting method [Friedman (2001)] for linear regression settings using an l2-loss function. The basis functions are simply the column vectors of the design matrix. l2boost scales the design matrix such that the boosting coefficients correspond to the gradient direction for each covariate. Friedman's gradient descent boosting algorithm proceeds at each step along the covariate direction closest (in L2 distance) to the maximal gradient descent direction.

The l2boost function uses an arbitrary L1-regularization parameter (nu), and includes the elementary data augmentation of Ehrlinger and Ishwaran (2012), to add an L2-penalization (lambda) similar to the elastic net [Zou and Hastie (2005)]. The L2-regularization reverses repressibility, a condition where one variable acts as a boosting surrogate for other, possibly informative, variables. Along with the decorrelation effect, this elasticBoost regularization circumvents L2Boost deficiencies in correlated settings.

We include a series of S3 functions for working with l2boost objects:

• print (print.l2boost) prints a summary of the l2boost model fit.

• coef (coef.l2boost) returns the model regression coefficients at any point along the solution path indexed by step m.

• fitted (fitted.l2boost) returns the fitted response values from the training set at any point along the solution path.

• residuals (residuals.l2boost) returns the training set residuals at any point along the solution path.

• plot (plot.l2boost) for graphing either beta coefficients or gradient-correlation as a function of boosting steps.

• predict (predict.l2boost) for boosting prediction on possibly new observations at any point along the solution path.

A cross-validation method (cv.l2boost) is also included for L2boost and elasticBoost cross-validating regularization parameter optimizations.

Example Datasets We have repackaged the diabetes data set from Efron et. al. (2004) for demonstration purposes. We also include data simulation functions for reproducing the elastic net simulation (elasticNetSim) of Zou and Hastie (2005) and the example multivariate normal simulations (mvnorm.l2boost) of Ehrlinger and Ishwaran (2012).

References

Friedman J. (2001) Greedy function approximation: A gradient boosting machine. Ann. Statist., 29:1189-1232

Ehrlinger J., and Ishwaran H. (2012). "Characterizing l2boosting" Ann. Statist., 40 (2), 1074-1101

Zou H. and Hastie T (2005) "Regularization and variable selection via the elastic net" J. R. Statist. Soc. B, 67, Part 2, pp. 301-320

Efron B., Hastie T., Johnstone I., and Tibshirani R. (2004). "Least Angle Regression" Ann. Statist. 32:407-499

ehrlinger/l2boost documentation built on April 28, 2022, 7:29 a.m.