setup_LINAD: Setup LINAD Hyperparameters

In rtemis: Schema-Driven Machine Learning and Visualization

Setup LINAD Hyperparameters

Description

Setup hyperparameters for LINAD, the Linear Additive Tree. LINAD generalizes the decision tree and the regularized linear model: CART, the Additive Tree and a penalized linear fit are each a special case of it, recovered exactly by the hyperparameter combinations given under Details.

Usage

setup_LINAD(
  max_leaves = 20L,
  min_cases_split = 2L,
  min_cases_leaf = 1L,
  min_cases_node_model = NULL,
  nvmax = NULL,
  forward_stop = NULL,
  lambda = NULL,
  alpha = NULL,
  learning_rate = 0.1,
  root_nvmax = NULL,
  root_lambda = NULL,
  root_alpha = NULL,
  root_learning_rate = 1,
  n_cuts = NULL,
  gamma = 0.1,
  split_binning = NULL,
  split_bin_type = "frequency",
  node_test = NULL,
  split_criterion = NULL,
  line_search = "expansion",
  constant_rule = "closed_form",
  ifw = FALSE,
  patience = NULL,
  force_max_leaves = FALSE,
  smooth_validation_curve = NULL,
  node_model = "forward",
  root_model = NULL,
  split_search = "stump",
  split_features = NULL,
  linear_features = NULL,
  global_features = NULL,
  line_search_max = 1000,
  root_forward_stop = NULL,
  root_node_test = NULL
)

Arguments

max_leaves	(Tunable) Integer [1, Inf): Largest number of terminal nodes to grow.
min_cases_split	(Tunable) Integer [2, Inf): Fewest cases a node may hold and still be considered for splitting.
min_cases_leaf	(Tunable) Integer [1, Inf): Fewest cases a split must leave on each side.
min_cases_node_model	(Tunable) Optional Integer [1, Inf): Fewest cases needed to fit a linear model at a node. Applies only when node_model is not "constant". NULL uses 10.
nvmax	(Tunable) Optional Integer [1, Inf): Most terms forward selection may add beside the intercept. Applies only when node_model is "forward". NULL uses 3.
forward_stop	(Tunable) Optional Character {"bic", "aic", "none"}: Cost a term must earn to be added by forward selection. Applies only when node_model is "forward". NULL uses "bic".
lambda	(Tunable) Optional Numeric [0, Inf): L2 penalty on the leaf models. Applies only when node_model is "forward", "ridge" or "elasticnet". NULL uses 0.05.
alpha	(Tunable) Optional Numeric [0, 1]: Elastic-net mixing, 0 ridge to 1 lasso. Applies only when node_model is "elasticnet". NULL uses 1.
learning_rate	(Tunable) Numeric (0, 1]: Shrinkage applied to every functional update.
root_nvmax	(Tunable) Optional Integer [1, Inf): nvmax for the root model. NULL uses nvmax.
root_lambda	(Tunable) Optional Numeric [0, Inf): lambda for the root model. NULL uses lambda.
root_alpha	(Tunable) Optional Numeric [0, 1]: alpha for the root model. NULL uses alpha.
root_learning_rate	(Tunable) Numeric [0, 1]: Shrinkage applied to the root model's slopes.
n_cuts	(Tunable) Optional Integer [2, Inf): Cut points tried per feature; split_bin_type decides their spacing. Applies only when split_search is "exhaustive". NULL uses 20.
gamma	(Tunable) Numeric [0, 1]: Weight a case retains in the branch it does not belong to. 0 is a hard partition.
split_binning	(Tunable) Optional Integer [2, Inf): Discretize each numeric feature into this many bins and consider only bin boundaries as splits. Applies to both split searches.
split_bin_type	(Tunable) Character {"frequency", "width"}: How bin edges are placed.
node_test	(Tunable) Optional Character {"none", "aic", "bic"}: Cost a node's slopes must earn over the constant alone. Applies only when node_model is "ridge" or "elasticnet"; forward selection uses forward_stop. NULL uses "none".
split_criterion	(Tunable) Optional Character {"mean", "linear"}: What the stump search scores a candidate side by. Applies only when split_search is "stump". NULL uses "mean".
line_search	(Tunable) Character {"expansion", "child", "none"}: Scope of the Newton step for each update.
constant_rule	(Tunable) Character {"closed_form", "least_squares"}: How a node's constant is computed. The two coincide for regression and differ only for classification.
ifw	(Tunable) Logical: If TRUE, use Inverse Frequency Weighting in classification.
patience	(Tunable) Optional Integer [1, Inf): Expansions without validation improvement before growth stops. Requires a validation set. NULL disables patience-based early stopping.
force_max_leaves	Logical: If TRUE, keep every leaf grown instead of selecting a size on the validation set.
smooth_validation_curve	Optional Logical: If TRUE, smooth the validation curve before reading its minimum. Applies only when force_max_leaves is FALSE. NULL does not smooth.
node_model	Character {"forward", "ridge", "elasticnet", "constant"}: Model fitted at each node. "constant" is the intercept-only model every decision tree node carries; the others add a regularized linear model on top of it.
root_model	Optional Character {"forward", "ridge", "elasticnet", "constant"}: Model fitted at the root. NULL uses node_model.
split_search	Character {"stump", "exhaustive"}: How a split is chosen.
split_features	Optional Character vector: Features that may define a split. NULL imposes no constraint, so every feature may.
linear_features	Optional Character vector: Features that get a slope in the node models. NULL imposes no constraint, so every feature does.
global_features	Optional Character vector: Linear features whose slope is shared by every leaf. Must be a subset of linear_features where that is set. NULL imposes no constraint, so no coefficient is pinned.
line_search_max	Numeric (0, Inf): Largest absolute step the line search may take.
root_forward_stop	(Tunable) Optional Character {"bic", "aic", "none"}: forward_stop for the root model. NULL uses forward_stop, or "bic" where that is NULL. Applies only when the root model is "forward".
root_node_test	(Tunable) Optional Character {"none", "aic", "bic"}: node_test for the root model. NULL uses node_test, or "none" where that is NULL. Applies only when the root model is "ridge" or "elasticnet".

Details

Every node carries a constant, as any decision tree's nodes do. node_model selects the model fitted at each node: "constant" for that constant alone, or a regularized linear model on top of it. That, with max_leaves and gamma, makes LINAD a generalization: the familiar algorithms are points in its hyperparameter space, and everything between them is reachable too.

Value

LINADHyperparameters object.

What LINAD generalizes

Each of the following is a special case of LINAD, recovered exactly:

A pure linear model

max_leaves = 1. Nothing is split, so the model is the root's alone – ridge, elastic net or forward stepwise selection, whichever node_model names. Verified against a direct penalized solve.

CART

node_model = "constant", gamma = 0, learning_rate = 1, line_search = "none". Constants only, a hard partition and no shrinkage. Fitted values match rpart's to machine precision.

The Additive Tree

The same, with gamma > 0. Soft weighting is precisely what separates the Additive Tree from CART (Luna and colleagues, 2019; see References).

LINAD

Any node_model other than "constant", with max_leaves > 1 – a tree whose nodes each carry a linear model.

Every setting named above is tunable, so the data can select where a dataset belongs in that space: a linear generating process should select a full root fit and no splitting, while terminal nodes needing opposite-signed coefficients should select a split first. root_learning_rate is the dial for that first step – at 0 the root is the outcome's mean and the tree splits first, at 1 a full linear model is fitted before any split.

How it grows

LINAD fits a regularized linear model on the whole sample, splits on the gradient of the loss, fits a model on each side of the split, and repeats. Every update is linear, so the coefficients along a root-to-leaf path sum: the finished model is a tree with one linear model per leaf, optimized stagewise rather than fitted leaf by leaf. Prediction routes a case to its leaf and evaluates that leaf's coefficients.

Splits are found on the features as given, so a factor splits on a set of its levels; the node models use a reference-coded design matrix built internally. LINAD therefore needs no encoding preprocessor and handles factors directly.

max_leaves plays the role that the number of trees plays in gradient boosting, and like it is chosen on held-out data: the tree is grown to max_leaves and the size with the lowest validation loss is kept, unless force_max_leaves is TRUE. Pass dat_validation to train to control the set that choice is made on.

gamma lets every case carry weight into both branches, decaying as gamma^depth, so each node's model is pulled toward what the rest of the data supports. It is the algorithm's main variance-reducing device.

split_search = "exhaustive" scores each candidate split by the loss after fitting both child models, rather than taking the best split of the gradient. It costs more per split and finds structure no stump can see: an interaction that changes another feature's slope without changing either side's mean leaves no trace in any one feature's marginal fit.

A node's model is fitted to what its parent left, so its slopes are the change to the parent's coefficients and lambda and alpha penalize that change. At alpha = 1 the change is a lasso, so a feature whose coefficient does not need to move gets exactly zero and inherits its parent's value: the tree decides per feature, per node, whether an effect keeps adapting. Ridge shrinks every change but zeroes none, so every coefficient moves at every node.

node_test lets the tree decide, node by node, whether a linear model is warranted at all: the slopes are kept only where they reduce the node's residual sum of squares by more than they cost, and the node takes a plain constant otherwise. A tree whose flat regions carry constants and whose structured regions carry coefficients is both smaller to read and a statement about where the response is locally linear. It buys that sparsity at a price in accuracy, so it is off by default.

It applies to the shrinking models only. Forward selection charges the same cost per term through forward_stop, and a set of terms that each paid for itself always pays for itself jointly, so a node-level test could never overturn it. One cost rule per node model, at the granularity that model admits.

split_criterion sets what the stump scores a side by, and closes part of that gap cheaply. "mean" scores the level its fit explains, which is the CART criterion and is blind to curvature – a parabola's two halves have the same mean by construction, so the correct split at the vertex scores near last. "linear" adds the slope in the split variable, at one further pass per feature per node, and on such a problem recovers the exhaustive search's fit. It cannot recover the cross-variable case above, and where the change belongs to another feature it fits the wrong one more confidently, so it is offered as a tunable rather than the default.

split_binning discretizes numeric features before either search, so a candidate split falls on a bin boundary rather than between any two distinct values; n_cuts then thins what is left, for the exhaustive search only. split_bin_type governs both: "frequency" spaces cuts evenly through the cases, "width" evenly through the feature's range.

Feature roles

A feature's behavior has two independent axes: whether it may define a partition, and whether its modeled effect is absent, shared by every subgroup, or free to change along a path. Three name-based selectors set them, and on each of the three NULL imposes no constraint – which means "all" for the first two and "nothing pinned" for the third:

split_features

Features the tree may split on.

linear_features

Features that receive a slope in the node models.

global_features

Linear features whose slope is shared by every leaf: the root estimates it and no node below may change it. Must be a subset of linear_features where that is set.

The axes are independent, so a feature may split without entering any node model, enter every node model without ever splitting, do both, or neither. Roles are assigned to source features: naming a factor moves all of its encoded columns together, never one of them.

Declaring effect scope turns LINAD into a varying-coefficient model with a hierarchical form – a global baseline plus the node deviations accumulated down the path. global is a hard constraint on the coefficient: it holds exactly, at any gamma. gamma is the soft, data-driven version of the same idea, pulling every node's fit toward what the rest of the data supports until, at gamma = 1, every coefficient is effectively global. The two compose without interacting: gamma decides how much a node's adaptive coefficients may move, and global_features decides which coefficients are allowed to move at all.

Because a global coefficient is estimated at the root, root_learning_rate shrinks it: at 0 the root fits no slopes and a global feature carries no effect anywhere.

The constrained family keeps useful reductions. No splits (max_leaves = 1) is a penalized linear model; node_model = "constant" is a tree of constants; making every linear feature global gives a tree with shared coefficients and node-specific constants; and leaving all three NULL reproduces unconstrained LINAD exactly.

Restricting the split set is also the largest single speed-up available on wide data, since the search cost is linear in the number of candidate features.

Learning rate and the root model

learning_rate shrinks the update made at each node. It does not shrink the root, which root_learning_rate governs and which defaults to 1: the root model is the initialization F_0, not a boosting step, and shrinking an initialization is not meaningful.

The consequence is worth stating, because it surprises: however small learning_rate is, the fit never falls below a single global linear model. On data whose signal is largely linear, that model is already good, so shrinking the tree's correction moves the metrics very little and two very different small learning rates look alike. On a signal the root cannot represent – a step with no linear component – the same sweep moves test R-squared from 0.58 to 0.97.

To shrink everything, including the root, set root_learning_rate to the same value. That recovers the familiar picture in which a small rate leaves the model badly under-fitted: at 1e-6 the fit reduces to the constant that alone minimizes the loss – the weighted mean of the outcome, or half its log odds for a classification – which scores an R-squared of 0 rather than anything worse. root_learning_rate interpolates between that constant and the full linear model, never towards zero.

The line_search step does not counteract this. It is scale-free – for squared error it is the exact minimizer along the update direction, which is near 1 whatever the residual's size – so the applied step stays proportional to learning_rate. Measured node updates scale linearly with it, by a factor of 50 across a 50-fold change in the rate.

line_search and constant_rule each select among alternative update rules – the scope of the Newton step, which frontier node is split next, and how a node's constant is computed. There is no settled best choice among them, so each is tunable.

Author(s)

EDG

References

Luna JM, Gennatas ED, Ungar LH, Eaton E, Diffenderfer ES, Jensen ST, Simone CB 2nd, Friedman JH, Solberg TD, Valdes G (2019). Building more accurate decision trees with the additive tree. Proceedings of the National Academy of Sciences, 116(40), 19887-19893. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1073/pnas.1816748116")}

Examples

linad_hyperparams <- setup_LINAD(max_leaves = 10L, learning_rate = 0.1)
linad_hyperparams
# The Additive Tree is LINAD without the linear models
addtree_hyperparams <- setup_LINAD(node_model = "constant", gamma = 0.8)
addtree_hyperparams

rtemis documentation built on Oct. 20, 2026, 1:06 a.m.