ag_matmul_precision: Matrix-multiply precision for ag_* operations on the GPU

In ggmlR: 'GGML' Tensor Operations for Machine Learning

Matrix-multiply precision for ag_* operations on the GPU

Description

On a Vulkan device with fp16 or cooperative-matrix support, an f32 x f32 matrix multiply with more than 8 columns runs a kernel that keeps its tiles in f16: relative error around 4e-4, while up to 8 columns (and the CPU) give f32 accuracy (~1e-7). "f32" asks every ag_matmul (forward, and the input gradient of the graph backward) for the f32-tile kernel, so the result no longer depends on the batch size or the device.

Usage

ag_matmul_precision(precision)

Arguments

precision

"default" or "f32". Missing: return the current setting.

Details

Cost: none measurable on small layers (RL policies, tens to hundreds of units); 2-4x slower on large ones (1024 x 1024 and up). That is why the default stays "default": set it around the code that needs it and restore the previous value, e.g. old <- ag_matmul_precision("f32"); on.exit(ag_matmul_precision(old)). Has no effect on the CPU or with ag_dtype("f16") / "bf16".

Value

The current setting when precision is missing; otherwise the previous setting, invisibly.

Examples

old <- ag_matmul_precision("f32")
ag_matmul_precision()
ag_matmul_precision(old)

ggmlR documentation built on Oct. 23, 2026, 1:07 a.m.