ggml_attention: Create a Multi-Head Attention Layer Object

In ggmlR: 'GGML' Tensor Operations for Machine Learning

Create a Multi-Head Attention Layer Object

Description

Returns a reusable layer object for use with ggml_apply, the scaled dot-product attention of Attention Is All You Need with n_heads heads computed in parallel. Applying the same object to several tensor nodes shares one set of projection weights, which is how an encoder block is reused across a stack.

Usage

ggml_attention(
  d_model,
  n_heads = 1L,
  causal = FALSE,
  bias = TRUE,
  rope = FALSE,
  rope_base = 10000,
  dropout = 0,
  name = NULL,
  trainable = TRUE
)

Arguments

d_model	Integer, model width. Must be divisible by n_heads and must match the parent's feature dimension.
n_heads	Integer, number of attention heads (default 1).
causal	Logical: mask out positions after the current one, so a query attends only to keys at or before it (default FALSE). This is what makes a decoder autoregressive. Only meaningful for self-attention – masking a cross-attention score matrix by position compares indices in two unrelated sequences, so it is rejected rather than silently applied.
bias	Logical: add a bias to the output projection (default TRUE).
rope	Logical: apply rotary position embedding to queries and keys (default FALSE). Weightless and relative, so it generalises to unseen sequence lengths. Needs an even d_model / n_heads.
rope_base	Rotation base for rope (default 10000).
dropout	Dropout rate applied to the attention weights during training (default 0, no dropout).
name	Optional character name.
trainable	Logical; whether weights are updated during training.

Details

For the pipe-style form that appends attention to a graph (x |> ggml_layer_attention(...)), see ggml_layer_attention.

Value

A ggml_layer object.

Shapes

The parent must be a sequence node of shape c(seq_len, d_model) – the same layout ggml_layer_lstm takes – and the output has that same shape, so attention blocks stack directly.

Self- and cross-attention

Applied to a single node the layer is self-attention: queries, keys and values all come from that node. Applied to a list of two nodes, ggml_apply(list(query, context), attn), it is cross-attention: queries come from the first, keys and values from the second, which may have a different sequence length (but must share d_model). This is the decoder-to-encoder path of a full transformer.

Weights

Four projections W_q, W_k, W_v, W_o, each d_model x d_model, plus an output bias b_o when bias = TRUE. Glorot-uniform initialised, zero bias.

See Also

ggml_layer_attention, ggml_apply

Examples

# One attention block shared between two sequences.
attn <- ggml_attention(d_model = 32L, n_heads = 4L)
x1   <- ggml_input(shape = c(10L, 32L))
x2   <- ggml_input(shape = c(10L, 32L))
o1   <- ggml_apply(x1, attn)
o2   <- ggml_apply(x2, attn)          # shared weights
stopifnot(identical(o1$layer_id, o2$layer_id))

# Cross-attention: queries from x1, keys/values from a longer context.
ctx  <- ggml_input(shape = c(15L, 32L))
o3   <- ggml_apply(list(x1, ctx), ggml_attention(32L, 4L))

ggmlR documentation built on Oct. 23, 2026, 1:07 a.m.