lr_scheduler_onecycle: One-cycle learning rate scheduler

In ggmlR: 'GGML' Tensor Operations for Machine Learning

One-cycle learning rate scheduler

Description

Implements the 1cycle policy (Smith, 2018): the learning rate rises from max_lr / div_factor to max_lr during the first pct_start fraction of training, then anneals down to max_lr / final_div_factor. Call $step() per batch; total_steps is the total number of batches over the whole run (epochs * batches_per_epoch).

Usage

lr_scheduler_onecycle(
  optimizer,
  max_lr,
  total_steps,
  pct_start = 0.3,
  div_factor = 25,
  final_div_factor = 10000,
  anneal_strategy = c("cos", "linear"),
  cycle_momentum = TRUE,
  base_momentum = 0.85,
  max_momentum = 0.95
)

Arguments

optimizer	Optimizer environment.
max_lr	Peak learning rate.
total_steps	Total number of $step() calls for the whole run.
pct_start	Fraction of total_steps spent increasing the LR (default 0.3).
div_factor	Initial LR is max_lr / div_factor (default 25).
final_div_factor	Final LR is max_lr / final_div_factor (default 1e4).
anneal_strategy	"cos" (default) or "linear".
cycle_momentum	Logical; cycle momentum inversely to LR (default TRUE).
base_momentum, max_momentum	Momentum bounds used when cycle_momentum = TRUE (defaults 0.85 / 0.95).

Details

When cycle_momentum = TRUE the momentum-like hyper-parameter moves inversely to the learning rate: $momentum for SGD, $beta1 for Adam. For other optimizers momentum cycling is skipped.

Value

An lr_scheduler_onecycle environment with $step() and $get_lr().

References

Smith, L. N. (2018). A disciplined approach to neural network hyper-parameters. arXiv:1803.09820.

Examples

w   <- ag_param(matrix(runif(4), 2, 2))
opt <- optimizer_sgd(list(w = w), lr = 0.01, momentum = 0.9)
sch <- lr_scheduler_onecycle(opt, max_lr = 0.1, total_steps = 100L)
for (i in 1:100) sch$step()

ggmlR documentation built on Oct. 23, 2026, 1:07 a.m.