nn_geglu: GeGLU Module

In mlr3torch: Deep Learning with 'mlr3'

GeGLU Module

Description

This module implements the Gaussian Error Linear Unit Gated Linear Unit (GeGLU) activation function. It computes \text{GeGLU}(x, g) = x \cdot \text{GELU}(g) where \(x\) and \(g\) are created by splitting the input tensor in half along the last dimension.

Usage

nn_geglu()

References

Shazeer N (2020). “GLU Variants Improve Transformer.” 2002.05202, https://arxiv.org/abs/2002.05202.

Examples

x = torch::torch_randn(10, 10)
glu = nn_geglu()
glu(x)

mlr3torch documentation built on Aug. 26, 2025, 5:09 p.m.