nn_geglu: GeGLU Module

In mlr-org/mlr3torch: Deep Learning with 'mlr3'

GeGLU Module

Description

This module implements the Gaussian Error Linear Unit Gated Linear Unit (GeGLU) activation function. It computes \text{GeGLU}(x, g) = x \cdot \text{GELU}(g) where \(x\) and \(g\) are created by splitting the input tensor in half along the last dimension.

Usage

nn_geglu()

References

Shazeer N (2020). “GLU Variants Improve Transformer.” 2002.05202, https://arxiv.org/abs/2002.05202.

Examples

x = torch::torch_randn(10, 10)
glu = nn_geglu()
glu(x)

mlr-org/mlr3torch documentation built on April 17, 2025, 8:22 p.m.