ag_layer_norm: Create a Layer Normalization layer

In ggmlR: 'GGML' Tensor Operations for Machine Learning

Create a Layer Normalization layer

Description

Normalizes each example independently over its normalized_shape features, then applies a learnable per-feature scale (gamma) and shift (beta): y = gamma * (x - mu) / sqrt(var + eps) + beta.

Usage

ag_layer_norm(normalized_shape, eps = 1e-05, elementwise_affine = TRUE)

Arguments

normalized_shape	Number of features to normalize over (the number of rows of the input).
eps	Small constant added to the variance for numerical stability (default 1e-5).
elementwise_affine	Logical; when TRUE (default) the layer learns gamma and beta. When FALSE it normalizes only and has no parameters.

Details

Unlike ag_batch_norm, which normalizes each feature across the batch, LayerNorm normalizes each column (example) across its features. It therefore has no running statistics and behaves identically in training and evaluation mode, which makes it the usual choice for transformers and for small or varying batch sizes.

The backward pass is exact: because mu and sigma are computed from the same example being normalized, their contribution to the gradient is included rather than treated as constant.

Value

An ag_layer_norm environment with $forward() and $parameters().

See Also

ag_batch_norm

Examples

ln <- ag_layer_norm(4L)
x  <- ag_param(matrix(rnorm(8), 4, 2))   # [features, batch]
with_grad_tape({
  y    <- ln$forward(x)
  loss <- ag_mse_loss(y, matrix(0, 4, 2))
})
grads <- backward(loss)

ggmlR documentation built on Oct. 23, 2026, 1:07 a.m.