lr_scheduler_cosine: Cosine-annealing learning rate scheduler

In ggmlR: 'GGML' Tensor Operations for Machine Learning

Cosine-annealing learning rate scheduler

Description

Varies the learning rate following a cosine curve from lr_max down to lr_min over T_max steps. Restarts (SGDR-style) if restart = TRUE.

Usage

lr_scheduler_cosine(optimizer, T_max, lr_min = 0, restart = FALSE, T_mult = 1)

Arguments

optimizer	Optimizer environment.
T_max	Number of steps for the first cosine cycle.
lr_min	Minimum learning rate (default 0).
restart	Logical; if TRUE restart after each cycle.
T_mult	Cycle-length multiplier applied after every restart (default 1, i.e. constant period). Only meaningful when restart = TRUE.

Details

With restart = TRUE and T_mult > 1 this is the full CosineAnnealingWarmRestarts (SGDR) schedule: the first cycle lasts T_max steps and every subsequent cycle is T_mult times longer than the previous one. T_mult = 1 (the default) keeps every cycle the same length.

Value

An lr_scheduler_cosine environment

References

Loshchilov, I. & Hutter, F. (2017). SGDR: Stochastic Gradient Descent with Warm Restarts. ICLR.

Examples

w   <- ag_param(matrix(runif(4), 2, 2))
opt <- optimizer_adam(list(w = w), lr = 0.1)
sch <- lr_scheduler_cosine(opt, T_max = 50L)
for (epoch in 1:50) sch$step()

# SGDR with doubling cycle length: 10, 20, 40, ...
opt2 <- optimizer_adam(list(w = w), lr = 0.1)
sch2 <- lr_scheduler_cosine(opt2, T_max = 10L, restart = TRUE, T_mult = 2)
for (epoch in 1:70) sch2$step()

ggmlR documentation built on Oct. 23, 2026, 1:07 a.m.