ggml_layer_transformer_block: Add a Transformer Encoder Block

In ggmlR: 'GGML' Tensor Operations for Machine Learning

Add a Transformer Encoder Block

Description

The standard pre-LN encoder block, in one call: normalize, attend, add the residual, normalize again, run the position-wise feed-forward, add the second residual. Composed from the existing layers – there is no new node type – so the pieces can still be wired by hand when a block needs to differ.

Usage

ggml_layer_transformer_block(
  x,
  d_model,
  n_heads = 1L,
  ff_dim = NULL,
  activation = "silu",
  norm = c("rms", "layer"),
  causal = FALSE,
  mask = NULL,
  rope = FALSE,
  dropout = 0,
  attn_dropout = dropout,
  name = NULL
)

Arguments

x	A ggml_tensor_node of shape c(seq_len, d_model).
d_model	Integer, model width. Must match the input's feature dimension and be divisible by n_heads.
n_heads	Integer, number of attention heads (default 1).
ff_dim	Integer, width of the feed-forward hidden layer. Defaults to 4 * d_model, the usual ratio.
activation	Feed-forward activation (default "silu"). "gelu" is trainable too, so a model ported from a GELU-based configuration keeps its activation. NB: "hardsigmoid" and "hardswish" still have no backward rule in ggml – the graph builds, then training aborts.
norm	"rms" (default) or "layer": which normalization to put in front of each sublayer.
causal	Logical: causal self-attention (default FALSE).
mask	Optional attention mask node, c(seq_q, seq_kv); see ggml_layer_attention.
rope	Logical: rotary position embedding on queries and keys (default FALSE).
dropout	Dropout rate applied to each sublayer's output before it is added back to the residual (default 0, meaning no dropout).
attn_dropout	Dropout rate for the attention weights specifically; defaults to dropout, so setting only dropout applies the same rate everywhere.
name	Optional prefix for the layers inside the block.

Value

A new ggml_tensor_node of shape c(seq_len, d_model), so blocks stack directly.

Pre-LN vs post-LN

Normalization comes BEFORE each sublayer, not after, which is what makes deep stacks trainable without a warmup schedule: the residual path stays un-normalized from input to output, so the gradient reaches the early layers undiminished. The original 2017 transformer normalized after; every large model since does it this way.

See Also

ggml_layer_attention, ggml_layer_sequence_pooling

Examples

# A two-block encoder with a pooled regression head.
x <- ggml_input(shape = c(10L, 32L))
h <- x |> ggml_layer_positional_embedding()
for (i in 1:2) h <- h |> ggml_layer_transformer_block(32L, n_heads = 4L)
y <- h |> ggml_layer_sequence_pooling() |> ggml_layer_dense(1L)

ggmlR documentation built on Oct. 23, 2026, 1:07 a.m.