get.beta: Local Coefficients and Term Contributions of a Super Greedy...

View source: R/utilities_beta.R

get.betaR Documentation

Local Coefficients and Term Contributions of a Super Greedy Forest

Description

Extracts the local regression coefficients (beta values) that a super greedy forest uses to predict each observation, and breaks each prediction down into the contributions of the individual terms. Works on the training data (a grow object) and on new data (a prediction object). An optional relaxed estimate replaces the lasso coefficients by least squares estimates computed from out-of-bag data. The helpers partial.beta and importance.beta use these coefficients to plot the partial effect of each variable and to compare it with the response.

Usage

get.beta(o, bag = c("oob", "inbag", "all")[1], papply = lapply, 
         relaxed = FALSE, active = c("global", "local"))

partial.beta(b, o, xvar.names = NULL, by = NULL, granule = 5,
             same.scale = TRUE, max.plots = 12, ...)

importance.beta(b, o)

Arguments

o

An object of class (rfsgt, grow) or (rfsgt, predict). For partial.beta and importance.beta, the object that was passed to get.beta, which holds the values of the variables. For importance.beta, new data must contain the response.

bag

Which trees to use for each training observation. The default "oob" uses the trees for which the observation is out-of-bag and reproduces the OOB predictions of the forest (predicted.oob). "inbag" uses the trees for which the observation is inbag and reproduces predicted. "all" uses every tree. For new data all trees are used, which is the same as bag = "all".

papply

Function used to loop over the trees, either lapply (default) or mclapply for parallel processing.

relaxed

Logical. If TRUE, the coefficients of each terminal node are re-estimated by least squares, without the lasso penalty, using the out-of-bag cases that fall in the node. See Details.

active

Terms used by the least squares estimate when relaxed = TRUE. "global" (default) uses all terms retained by the forest, the same set in every node. "local" uses only the terms with a nonzero lasso coefficient in the node.

b

The value of get.beta, with or without relaxed = TRUE. For importance.beta it must be computed without relaxed = TRUE and, for the training data, with bag = "oob" (the default).

xvar.names

Names of the variables to plot. By default all variables with a nonzero partial effect are plotted, ordered by the standard deviation of the effect, up to max.plots of them.

by

Optional name of a second variable. The observations are divided into at most three groups of this variable, and each group gets its own curve. See Details.

granule

A variable with at most this many distinct values is shown by boxplots instead of a scatter plot.

same.scale

Logical. If TRUE, all panels use the same vertical scale, which shows which variables have the larger effects. The scale covers the central 99 percent of the effects.

max.plots

Maximum number of panels when xvar.names is not specified.

...

Further graphical parameters passed to plot.

Details

The terminal node of a super greedy tree contains the results from a lasso fit to the inbag cases (in the case of a lasso failure, standard CART prediction is used). get.beta collects the polynomials of these successful lasso models for each observation and averages their coefficients over the trees. The result, beta, contains one set of local regression coefficients per observation. These coefficients describe how the prediction of the forest depends on each term near that observation, which can change from one region of the feature space to another.

get.beta also breaks each prediction down by term. The contribution of a term is its coefficient times the value of the term for the observation, for example \beta_1 x_1 for a main effect or \beta_{12} x_1 x_2 for an interaction. The contributions are averaged over trees and returned in partial, whose first column is the intercept. Each row of partial adds up to the prediction, apart from an adjustment for the restricted score described below. Plotting the contribution of a term against its variable shows the partial effect of that term, and the column means of beta summarize which terms the forest uses overall.

Interpreting these coefficients needs some care. The lasso shrinks them toward zero, especially in deep trees. Growing shallow trees (a small treesize) reduces the shrinkage, and relaxed = TRUE further mitigates this problem by re-estimating the coefficients by least squares on out-of-bag data. Both options are explained below.

Which trees are averaged

The option bag decides which trees are used for the calculations. With the default bag = "oob", only the trees for which the observation is out-of-bag are used, and the reconstructed prediction predicted equals the OOB prediction of the forest (predicted.oob). With bag = "inbag" it equals the inbag prediction (predicted), and bag = "all" uses every tree.

For new data, every tree used by the prediction contributes (all trees, or the trees chosen with get.tree in predict.rfsgt), and predicted equals the prediction stored in the prediction object.

A tree can also contain CART terminal nodes, which the algorithm uses in place of lasso nodes where the lasso becomes unstable (see rfsgt). A CART terminal node predicts with the mean response of its inbag cases. get.beta treats this mean as the intercept and sets all other coefficients of the node to zero.

Coefficients in the original units

With node.center = TRUE (the default in rfsgt), each node estimates its polynomial after centering the variables at the node. Coefficients on this centered scale depend on the node center and cannot be compared or averaged across nodes. get.beta therefore rewrites each node polynomial in terms of the original, uncentered variables before averaging, and all coefficients and contributions are reported on the scale of the data.

Rewriting a centered square or product produces additional lower order terms. For example, a product term in a node with centers c_1 and c_2 expands as

(x_1 - c_1)(x_2 - c_2) = x_1 x_2 - c_2 x_1 - c_1 x_2 + c_1 c_2.

The expansion contains the main effects x_1 and x_2, and the constant is added to the intercept. For this reason beta and partial can have more columns than the forest has retained terms. In this example they include columns for X1 and X2 even if the forest retained only the product X1_I_X2.

The columns are labeled by the dictionary terms. The first column of beta (labeled betaZ) is the intercept. A square ends in _SQ2_, as in X3_SQ2_ for the square of X3, and a product joins the two variable names with _I_, as in X1_I_X2 for the product of X1 and X2.

Restricted score

When the forest uses the restricted score (node.restrict = TRUE, the default in rfsgt), out-of-bag and new observations are evaluated in two steps (see predict.rfsgt). The node polynomial is evaluated at the nearest point of the box spanned by the inbag cases of the node, and the result is then confined to the range of the inbag responses. The contributions in partial are computed at that nearest point. The second step is not part of the polynomial, and get.beta reports its effect separately as response.adjustment. With this adjustment included, rowSums(partial) + response.adjustment equals predicted exactly.

Shrinkage of the coefficients

In deep trees lasso coefficients are biased toward zero for two reasons. First, the lasso penalty, chosen by cross-validation within the node, shrinks the coefficients. In a small node a square or a product varies little, and the penalty tends to drop these terms altogether. Second, the splits that formed the node were chosen using the same responses that are then used to estimate its polynomial, which attenuates the coefficients further. Note that this biasing does not degrade prediction, it is merely an artifact of penalization/regularization.

When the true regression function is a polynomial (or can be approximated by the dictionary terms), the column means of beta estimate the coefficients. In our runs of the example below, with about 100 inbag cases per terminal node the column means were within 15 percent of the true values for the linear and square terms and within 25 percent for the interaction. With about 30 cases per node (treesize = 10) they fell to about half the true values, while the prediction error stayed the same. If the coefficients are of primary interest, grow shallow trees by setting a small treesize. The default tree size of rfsgt is chosen for prediction, and its terminal nodes are too small to estimate the coefficients well.

Relaxed estimation

Setting relaxed = TRUE keeps the terms of each node but replaces their lasso coefficients by ordinary least squares estimates, as in the relaxed lasso with relaxation parameter zero. In each terminal node, least squares uses the out-of-bag cases of that tree that fall in the node. With the default bsf = "inbag", these cases played no part in growing the tree, and the estimates are free of both the lasso shrinkage and the attenuation caused by the splits. In our runs of the example below, the relaxed coefficients were within 5 percent of the true values for trees with 3 and 5 terminal nodes, and the relaxed polynomials predicted as well as the forest.

The argument active decides which terms enter the least squares estimate. With active = "global", every node uses all terms retained by the forest. With active = "local", each node uses only the terms its own lasso kept. The local version has fewer coefficients and needs fewer cases, but a term that the node lasso dropped stays out.

Least squares needs enough data. A node is used only if its number of out-of-bag cases exceeds its number of coefficients (the intercept plus the active terms) by at least five, and the other nodes are skipped. Shallow trees, and the large nodes of deeper trees, meet this requirement. The component nodes.used reports the fraction of nodes that were used, and nodes gives the number of nodes averaged for each observation. An observation that falls only in skipped nodes gets NA.

A term that is an exact linear combination of the other active terms within a node, such as the square of a binary variable, cannot be estimated there. Least squares drops it from that node and its coefficient in the node is zero. The component aliased reports for each term the fraction of used nodes in which this happened.

The relaxed polynomials are evaluated directly, without the restricted score, and response.adjustment is zero. For a training observation, predicted is a leave-one-out prediction. In each node where the observation was one of the out-of-bag cases used by least squares, its value is computed from the estimate with that observation left out, and the error of predicted is therefore an out-of-sample error. The in-sample values are in fitted, and the contributions in partial add up to fitted. For new data, predicted and fitted are the same.

The standard errors in se come from the least squares estimates. If a coefficient is averaged over B nodes whose least squares standard errors are s_1, \ldots, s_B, its reported standard error is

\sqrt{s_1^2 + \cdots + s_B^2} / B,

which is the standard error of an average of B independent estimates. To check whether a coefficient varies over the feature space, compare its spread over the observations with its standard error. If the standard deviation of a column of beta is no larger than its typical standard error, the data give no evidence that the coefficient changes from one region to another.

Partial effect plots

partial.beta plots the partial effect of each original variable from the value of get.beta. A variable can enter several terms, namely its linear term, its powers and its products with other variables, and its partial effect combines all of them. For an observation with local coefficients \beta, the partial effect of a variable x is the value of its local polynomial minus the value of the same polynomial with x set to its mean, while the other variables keep their observed values. For example, for the polynomial

\beta_1 x + \beta_{11} x^2 + \beta_{12} x z

the partial effect of x is

\beta_1 (x - m) + \beta_{11} (x^2 - m^2) + \beta_{12} (x - m) z,

where m is the mean of x. Terms that do not involve x cancel. Unlike a sum of the contributions in partial, the partial effect does not depend on where the origin of a variable lies. For a binary variable or a factor, whose mean is not one of its values, the average of the polynomial over the values of x in the data replaces the polynomial at the mean.

Each panel plots the partial effect against the variable and adds a smooth curve. The points form a band around the curve because the coefficients are local and because product terms involve other variables. A variable with at most granule distinct values is shown by boxplots. A factor is treated as one variable, and its effect combines the terms of all its levels. Levels that the filter did not retain are shown together as other. By default the panels are ordered by the standard deviation of the effect and use the same vertical scale.

With by, the observations are divided into groups of a second variable and each group gets its own curve. The groups are the values of the variable if it has at most three, and otherwise its low, middle and high thirds. Curves with different slopes or shapes indicate an interaction between the two variables.

The coefficients can come from the lasso or from the relaxed estimate. Partial effects computed from lasso coefficients inherit their shrinkage.

Importance from the partial effects

importance.beta compares the partial effect of each variable with the response. Let y be the response, r the response minus predicted, and e the partial effect of the variable, with y and e centered at their means. The marginal value is

\frac{\mathrm{mean}\,(y - e)^2}{\mathrm{mean}\,y^2},

which compares the partial effect of the variable alone with the response. The conditional value is

1 - \frac{\mathrm{mean}\,(r + e)^2 - \mathrm{mean}\,r^2}{\mathrm{mean}\,y^2},

one minus the increase in the standardized prediction error when the partial effect is removed from the prediction. Both values equal 1 for a variable with no term in the forest and are smaller for a larger effect. The marginal value is the smaller of the two when the partial effect is positively correlated with the rest of the prediction, as for correlated variables with effects of the same sign.

Value

A list with the following components.

beta

Matrix of local coefficients with one row per observation. Each entry is the coefficient of a term, averaged over the terminal nodes the observation falls in and expressed on the original scale of the data. The first column (betaZ) is the intercept.

betaZ

Matrix with one row per observation and one column per retained term, describing the CART terminal nodes. When an observation falls in a CART terminal node, the node mean is recorded for each term used in the split that created the node, and zero for the other terms. These values are averaged over the trees in which the observation falls in a CART terminal node. NA if the observation falls only in lasso terminal nodes, and NULL when relaxed = TRUE.

lasso.percent

Percentage of the terminal nodes in the forest that are lasso nodes. The rest are CART terminal nodes.

predicted

Prediction reconstructed from the terminal node models. It equals the forest prediction for the trees selected by bag, for example predicted.oob of the grow object when bag = "oob". When relaxed = TRUE, it is the prediction of the relaxed polynomials, computed by leave-one-out for the training data.

partial

Matrix of term contributions with the same layout as beta. Each entry is a coefficient times the value of its term, averaged over trees. The first column is the intercept.

response.adjustment

Change in the prediction caused by confining it to the inbag response range of the node, averaged over trees. Zero when the restricted score is off or never takes effect, and when relaxed = TRUE.

When relaxed = TRUE, the list also contains the following components.

fitted

In-sample values of the relaxed polynomials, equal to rowSums(partial).

se

Matrix of standard errors for the coefficients in beta.

nodes

Number of terminal nodes averaged for each observation.

nodes.used

Fraction of the terminal nodes with enough out-of-bag cases for least squares.

aliased

Fraction of the used nodes in which each term was an exact linear combination of the other active terms and was dropped from the least squares estimate.

relaxed

List with the settings used, namely active and min.extra (the number of out-of-bag cases a node needs beyond its number of coefficients).

partial.beta is called for its plot. It returns, invisibly, a list with the following components.

effect

Matrix of partial effects with one row per observation and one column per plotted variable.

spread

Standard deviation of each partial effect over the observations, in the units of the response.

group

Factor with the group of each observation when by is used, and NULL otherwise.

importance.beta returns a data frame with one row per original variable and the columns marginal and conditional defined in Details, ordered by marginal. A variable that the filter removed has the value 1 in both columns.

Author(s)

Hemant Ishwaran and Udaya B. Kogalur

See Also

rfsgt, predict.rfsgt, vimp.rfsgt

Examples


## ------------------------------------------------------------
## local coefficients and partial effects
## ------------------------------------------------------------

## The true model is y = 2 x1 - x2 + 1.5 x1 x2 + x3^2 + noise, and x4 and
## x5 have no effect.  With hcut = 3 the model terms include all linear,
## square and pairwise product terms, which lets the terminal node
## polynomials represent the true model.
set.seed(1106)
n <- 500
d <- data.frame(matrix(runif(n * 5, -1, 1), n, 5))
d$y <- with(d, 2 * X1 - X2 + 1.5 * X1 * X2 + X3^2) + rnorm(n, sd = 0.25)

## Keep all 20 terms (filter = FALSE), use lasso terminal nodes
## everywhere (pure.lasso = TRUE), and grow shallow trees with 3 terminal
## nodes.  Each node then holds about 100 inbag cases, enough to estimate
## the coefficients well.
o <- rfsgt(y ~ ., d, hcut = 3, filter = FALSE, pure.lasso = TRUE, treesize = 3)
print(o)

## Coefficients and contributions behind the OOB predictions (the
## default bag = "oob").  The reconstructed predictions match the forest.
b <- get.beta(o)
print(all.equal(b$predicted, o$predicted.oob))

## One row per observation and one column per term.  betaZ is the
## intercept, X3_SQ2_ is the square of X3, and X1_I_X2 is the product of
## X1 and X2.
print(round(head(b$beta), 2))

## The column means estimate the true coefficients of x1, x2, x1 x2 and
## x3^2 (2, -1, 1.5 and 1), shrunk somewhat toward zero by the lasso, and
## the means of the other terms are near zero.  Deeper trees shrink the
## coefficients more (try treesize = 10).
print(round(sort(colMeans(b$beta[, -1], na.rm = TRUE), decreasing = TRUE), 2))

## Each contribution is a coefficient times the value of its term.  The
## contributions plus the response adjustment add up to the prediction.
print(round(head(b$partial), 2))
print(all.equal(rowSums(b$partial) + b$response.adjustment, b$predicted))

## Partial effects, shown by plotting the contribution of a term against
## its variable.
oldpar <- par(mfrow = c(1, 3))
plot(d$X1, b$partial[, "X1"], xlab = "x1", ylab = "contribution of x1")
plot(d$X3, b$partial[, "X3_SQ2_"], xlab = "x3", ylab = "contribution of x3^2")
plot(d$X1 * d$X2, b$partial[, "X1_I_X2"], xlab = "x1 x2",
     ylab = "contribution of x1 x2")
par(oldpar)

## Prediction example
new <- data.frame(matrix(runif(5 * 5, -1, 1), 5, 5))
p <- predict(o, new)
bn <- get.beta(p, bag = "all")
print(round(cbind(predicted = p$predicted, bn$partial), 2))

## Relaxed estimation by least squares on the OOB cases of each node.
## Without the lasso penalty and the selection effect of the splits, the
## column means are close to the true coefficients.
r <- get.beta(o, relaxed = TRUE)
print(round(sort(colMeans(r$beta[, -1], na.rm = TRUE), decreasing = TRUE), 2))
cat("fraction of nodes with enough OOB cases:", round(r$nodes.used, 2), "\n")

## OOB error of the forest compared with the leave-one-out error of the
## relaxed polynomials.
c(forest = mean((d$y - o$predicted.oob)^2, na.rm = TRUE),
  relaxed = mean((d$y - r$predicted)^2, na.rm = TRUE))

## Spread of the x1 coefficient over the observations compared with its
## standard error.  The true coefficient is constant and equal to 2.
print(c(sd = sd(r$beta[, "X1"], na.rm = TRUE), se = median(r$se[, "X1"], na.rm = TRUE)))

## Partial effect plots of the variables, here from the relaxed
## coefficients.  The effect of a variable combines all terms that
## involve it.  The true effects are a line of slope 2 for X1, a line of
## slope -1 for X2 and a parabola for X3, and X4 and X5 have no effect.
pe <- partial.beta(r, o)
print(round(pe$spread, 2))

## Effect of X1 for low, middle and high values of X2.  The product term
## 1.5 x1 x2 gives slopes near 1, 2 and 3.
partial.beta(r, o, "X1", by = "X2")

## Partial effect importance of each variable compared with the response, from the
## OOB lasso coefficients.  X4 and X5 have no effect and values near 1.
print(round(importance.beta(b, o), 3))


randomForestSGT documentation built on Nov. 16, 2026, 5:07 p.m.