setup_MLP: Setup MLP Hyperparameters

In rtemis: Schema-Driven Machine Learning and Visualization

Setup MLP Hyperparameters

Description

Setup hyperparameters for MLP (Multilayer Perceptron) training.

Usage

setup_MLP(
  hidden_units = NULL,
  shape = NULL,
  shape_layers = NULL,
  shape_max_units = NULL,
  activation = "relu",
  norm = NULL,
  residual = FALSE,
  dropout = 0,
  input_dropout = 0,
  weight_decay = 0,
  l1_penalty = 0,
  embedding_dim = NULL,
  embedding_dropout = 0,
  optimizer = "adamw",
  lr = 0.001,
  batch_size = 256L,
  max_epochs = 100L,
  max_grad_norm = NULL,
  ifw = FALSE,
  norm_first = FALSE,
  bias = TRUE,
  embeddings = TRUE,
  beta1 = NULL,
  beta2 = NULL,
  eps = NULL,
  momentum = NULL,
  lr_scheduler = NULL,
  patience = 10L,
  loss = NULL,
  seed = NULL,
  num_workers = 0L,
  drop_last = FALSE
)

Arguments

hidden_units	(Tunable) Optional Integer [1, Inf) vector: Units in each hidden layer, one value per layer. NULL generates the widths from the shape settings.
shape	(Tunable) Optional Character {"funnel", "constant", "triangle", "long_funnel", "diamond", "hexagon", "stairs"}: Profile of the generated hidden layer widths. Ignored when hidden_units is set. NULL uses "funnel" where hidden_units is NULL.
shape_layers	(Tunable) Optional Integer [1, Inf): Number of hidden layers to generate. Ignored when hidden_units is set. NULL uses 3 where hidden_units is NULL.
shape_max_units	(Tunable) Optional Integer [1, Inf): Widest generated hidden layer. Ignored when hidden_units is set. NULL derives it from the encoded input width where hidden_units is NULL.
activation	(Tunable) Character {"relu", "gelu", "silu", "elu", "selu", "leaky_relu", "tanh"}: Activation applied after every hidden layer.
norm	(Tunable) Optional Character {"batch_norm", "layer_norm"}: Normalization applied in every hidden layer. NULL applies none.
residual	(Tunable) Logical: If TRUE, add a residual connection around every hidden layer, projected when the layer changes width.
dropout	(Tunable) Numeric [0, 1): Dropout probability applied after every hidden layer.
input_dropout	(Tunable) Numeric [0, 1): Dropout probability applied to the encoded input.
weight_decay	(Tunable) Numeric [0, Inf): L2 penalty, decoupled from the gradient under the adamw optimizer.
l1_penalty	(Tunable) Numeric [0, Inf): L1 penalty on the linear weights, added to the loss.
embedding_dim	(Tunable) Optional Integer [1, Inf): Width of every embedding. NULL sizes each from its feature's cardinality.
embedding_dropout	(Tunable) Numeric [0, 1): Dropout probability applied to the concatenated embeddings.
optimizer	(Tunable) Character {"adamw", "adam", "sgd", "rmsprop"}: Optimization algorithm.
lr	(Tunable) Numeric (0, Inf): Learning rate.
batch_size	(Tunable) Integer [1, Inf): Cases per optimization step.
max_epochs	(Tunable) Integer [1, Inf): Largest number of passes over the training set.
max_grad_norm	(Tunable) Optional Numeric (0, Inf): Clip the gradient norm to this value before each step. NULL does not clip.
ifw	(Tunable) Logical: If TRUE, use Inverse Frequency Weighting in classification.
norm_first	Logical: If TRUE, normalize before the activation rather than after it.
bias	Logical: If TRUE, give every hidden layer and the output layer a bias term.
embeddings	Logical: If TRUE, represent each categorical feature by a learned embedding; if FALSE, one-hot encode them.
beta1	Optional Numeric [0, 1): Exponential decay rate of the first moment estimate. Applies to the adam and adamw optimizers.
beta2	Optional Numeric [0, 1): Exponential decay rate of the second moment estimate. Applies to the adam and adamw optimizers.
eps	Optional Numeric (0, Inf): Term added to the denominator for numerical stability. Applies to the adam, adamw and rmsprop optimizers.
momentum	Optional Numeric [0, Inf): Momentum factor. Applies to the sgd and rmsprop optimizers.
lr_scheduler	Optional Character {"step", "cosine_annealing", "one_cycle", "reduce_on_plateau"}: Learning-rate schedule. NULL holds the learning rate fixed.
patience	Integer [1, Inf): Epochs without validation improvement before stopping early.
loss	Optional Character {"mse", "l1", "smooth_l1", "cross_entropy"}: Training objective. NULL sets it from the outcome type.
seed	Optional Integer: Random seed for weight initialization, dropout and batch shuffling.
num_workers	Integer [0, Inf): Subprocesses used to load batches.
drop_last	Logical: If TRUE, drop the last incomplete batch of each training epoch.

Details

A fully connected feedforward network built and trained with torch, for regression, binary and multiclass classification.

Architecture. Give the hidden layers directly with hidden_units – c(256L, 128L, 64L) is three layers of those widths – or leave it NULL and let shape, shape_layers and shape_max_units generate them. Setting both is an error. The generated profiles are funnel (a linear taper from the widest layer down to a third of it), constant, triangle (a linear rise from the input width), long_funnel, diamond, hexagon and stairs; the vocabulary is Talos's, by way of AutoPyTorch. shape_max_units defaults to four times the encoded input width, clamped to [64, 512] and never below that width.

The tabular deep-learning benchmarks tune and publish constant-width MLPs, so shape = "constant" is what to compare against even though funnel is the better default before tuning on the small-n, wide-p data rtemis is usually pointed at.

Tuning. hidden_units is tunable like any other hyperparameter, with one architecture per candidate: setup_MLP(hidden_units = tune_over(c(64L, 32L), c(128L, 64L, 32L))). A single bare vector there is one architecture, not a set of candidates, and is rejected as such.

Categorical features are represented by learned embeddings, each sized min(600, round(1.6 * cardinality^0.56)) unless embedding_dim fixes them all. embeddings = FALSE one-hot encodes them. Numeric features are always centered and scaled, and the fitted encoder is re-applied at predict time.

Device and reproducibility. The device is the execution config's: see the device argument of setup_FutureExecution. Unset picks cuda where available and cpu otherwise, and train names the one it resolved. "mps", the Apple silicon GPU, runs only when named. In rtemis benchmarks on an Apple M5 the CPU was faster for every network narrower than about 1024 units or trained in batches smaller than about 2048 cases, often by 2 to 5 times; with both, and thousands of features, "mps" was faster by up to about 25%. It also slows down across many fits in one R session. Try it for a wide network on wide data. A seed governs weight initialization and batch shuffling on mps but not dropout, so a seeded mps fit reproduces exactly until a dropout rate is non-zero. That combination warns.

Early stopping needs a validation set: pass dat_validation to train, and the fit keeps the weights of the best validation epoch rather than the last. Without one it runs the full max_epochs and patience has no effect.

l1_penalty is not weight_decay. weight_decay is the L2 term the torch optimizer applies, decoupled from the gradient under adamw; l1_penalty has no torch equivalent and is accumulated over the linear weights and added to the loss.

Batch normalization and dropout interact badly when combined; both are off by default.

The scheduler takes no settings of its own – each configures itself from the run's budget: step decays by 0.1 every max_epochs / 3 epochs, cosine_annealing anneals over max_epochs, one_cycle peaks at lr over the run's real step count, and reduce_on_plateau decays by 0.1 after half the early-stopping patience.

get_varimp() returns NULL: a torch MLP has no native importance measure.

Value

MLPHyperparameters object.

Author(s)

EDG

Examples

mlp_hyperparams <- setup_MLP(hidden_units = c(64L, 32L), max_epochs = 20L)
mlp_hyperparams

rtemis documentation built on Oct. 20, 2026, 1:06 a.m.