optimizer_adam: Create an Adam optimizer

In ggmlR: 'GGML' Tensor Operations for Machine Learning

Create an Adam optimizer

Description

Create an Adam optimizer

Usage

optimizer_adam(
  params,
  lr = 0.001,
  beta1 = 0.9,
  beta2 = 0.999,
  eps = 1e-08,
  accumulate_steps = 1L,
  average = TRUE
)

Arguments

params	List of ag_param tensors, named or not (names are used only for display). The same tensor listed twice is kept once, with a warning.
lr	Learning rate (default 1e-3)
beta1	First moment decay (default 0.9)
beta2	Second moment decay (default 0.999)
eps	Stability constant (default 1e-8)
accumulate_steps	Number of backward() passes to accumulate before a call to $step() updates the weights (default 1, no accumulation). The bias-correction counter advances only on a real update, so accumulating k micro-batches behaves like one step on the larger batch rather than like k steps.
average	When accumulating, divide the accumulated gradient by accumulate_steps (default TRUE). See optimizer_sgd.

Value

An optimizer environment with step(grads), zero_grad(), state() and load_state(state). state() returns a versioned list (moments as host double matrices, t, accum_count); load_state() checks the type, parameter count, names (when both sides have them) and shapes before writing anything, restores t and accum_count, and does NOT restore hyperparameters (it warns when they differ). Exact continuation holds on the same device and dtype; CPU <-> GPU in f32 continues within float tolerance; across f16 and f32 it cannot be exact. Only read saved states you trust (readRDS).

On the GPU the step always runs in F32, whatever ag_dtype is: the moments are F32, the gradient is converted to F32 before it is squared, and a weight held in f16 has an F32 master copy that the step updates and then copies into the f16 weight, so updates smaller than the f16 spacing still accumulate. state() includes these master weights, and load_state() installs them as the parameter values; load the optimizer state after the model's own weights.

Examples

w <- ag_param(matrix(runif(4), 2, 2))
opt <- optimizer_adam(list(w = w), lr = 1e-3)

ggmlR documentation built on Oct. 23, 2026, 1:07 a.m.