ggml_layer_attention: Add a Multi-Head Attention Layer

In ggmlR: 'GGML' Tensor Operations for Machine Learning

Add a Multi-Head Attention Layer

Description

Scaled dot-product attention with n_heads heads, in the pipe-style form that appends a layer to a functional graph. For the reusable layer object that shares one set of weights between several applications – how an encoder block is reused across a stack – see ggml_attention.

Usage

ggml_layer_attention(
  x,
  d_model,
  n_heads = 1L,
  causal = FALSE,
  context = NULL,
  mask = NULL,
  rope = FALSE,
  rope_base = 10000,
  dropout = 0,
  bias = TRUE,
  name = NULL,
  trainable = TRUE
)

Arguments

x	A ggml_tensor_node of shape c(seq_len, d_model), or a list of two such nodes list(query, context) for cross-attention.
d_model	Integer, model width. Must be divisible by n_heads and match the parent's feature dimension.
n_heads	Integer, number of attention heads (default 1).
causal	Logical: mask out positions after the current one, making the layer autoregressive (default FALSE). Self-attention only.
context	Optional ggml_tensor_node to take keys and values from, making the layer cross-attention: x supplies the queries and decides the output length, context the keys and values. The two may differ in sequence length but must share d_model. Equivalent to passing list(query, context) as x.
mask	Optional ggml_tensor_node of shape c(seq_q, seq_kv) – one row per query, holding that query's entry for every key – added to the scores before the softmax: use 0 where a key may be attended to and a large negative value (-1e9) where it may not. This is how padding is excluded when a batch holds sequences of different lengths. Combines with causal.
rope	Logical: apply rotary position embedding to the queries and keys (default FALSE). Unlike ggml_layer_positional_embedding this has no weights and encodes RELATIVE position – the score between two tokens depends on the distance between them – so it carries over to sequence lengths the model was not trained on. Needs an even d_model / n_heads.
rope_base	Rotation base for rope (default 10000, the value
dropout	Dropout rate applied to the attention weights during training (default 0, no dropout). used by LLaMA-family models). Larger values stretch the rotation over longer contexts.
bias	Logical: add a bias to the output projection (default TRUE).
name	Optional layer name.
trainable	Logical.

Value

A new ggml_tensor_node of shape c(seq_len, d_model).

Shapes

x must be a sequence node of shape c(seq_len, d_model), the same layout ggml_layer_lstm takes. The output has that same shape, so attention blocks stack directly and a residual connection (ggml_layer_add(list(x, attn_out))) needs no reshaping.

How it is computed

All heads are computed in one batched pass rather than in a loop: the projections are reshaped to [d_head, n_heads, seq, batch] and permuted so that ggml_mul_mat() batches over the head and batch axes. The node count is therefore independent of n_heads.

Cross-attention

Pass a list of two nodes, ggml_layer_attention(list(query, context), ...), to take queries from the first and keys/values from the second. The two may differ in sequence length but must share d_model. A single node is self-attention.

See Also

ggml_attention for the shared-weights layer object.

Examples

# A transformer encoder block: attention + residual, then a feed-forward
# sublayer + residual.
x    <- ggml_input(shape = c(10L, 32L))
attn <- x |> ggml_layer_attention(d_model = 32L, n_heads = 4L)
h    <- ggml_layer_add(list(x, attn))
ff   <- h |> ggml_layer_dense(32L, activation = "relu")
out  <- ggml_layer_add(list(h, ff))

# GPT-style causal self-attention.
dec <- x |> ggml_layer_attention(32L, n_heads = 4L, causal = TRUE)

# Padding mask, fed as a second input alongside the sequence.
pad <- ggml_input(shape = c(10L, 10L), name = "mask")
enc <- x |> ggml_layer_attention(32L, n_heads = 4L, mask = pad)

ggmlR documentation built on Oct. 23, 2026, 1:07 a.m.