lr_scheduler_warmup_cosine: Linear warmup followed by cosine annealing

In ggmlR: 'GGML' Tensor Operations for Machine Learning

Linear warmup followed by cosine annealing

Description

Raises the learning rate linearly from warmup_start_lr to the optimizer's initial lr over warmup_steps steps, then anneals it to lr_min following a cosine curve over the remaining total_steps - warmup_steps steps. This is the standard transformer training schedule. Call $step() per batch.

Usage

lr_scheduler_warmup_cosine(
  optimizer,
  warmup_steps,
  total_steps,
  lr_min = 0,
  warmup_start_lr = 0
)

Arguments

optimizer	Optimizer environment. Its current lr is taken as the peak learning rate.
warmup_steps	Number of linear warmup steps.
total_steps	Total number of $step() calls (warmup included).
lr_min	Final learning rate after annealing (default 0).
warmup_start_lr	Learning rate at step 0 (default 0).

Value

An lr_scheduler_warmup_cosine environment with $step() and $get_lr().

Examples

w   <- ag_param(matrix(runif(4), 2, 2))
opt <- optimizer_adam(list(w = w), lr = 1e-3)
sch <- lr_scheduler_warmup_cosine(opt, warmup_steps = 10L, total_steps = 100L)
for (i in 1:100) sch$step()

ggmlR documentation built on Oct. 23, 2026, 1:07 a.m.