proj_add_norm: Project, Add, and Normalize

In macmillancontentscience/torchtransformers: Transformer Models in Torch

Project, Add, and Normalize

Description

Takes in two tensors, an "input" and a "residual". Applies a linear projector to the input (changing the size to match residual), performs dropout, adds the result to the residual, then applies layer normalization to the sum.

Usage

proj_add_norm(input_size, output_size, hidden_dropout = 0.1)

Arguments

input_size	Integer; the size of input tensor.
output_size	Integer; the size of output tensor (must match residual).
hidden_dropout	Numeric; dropout probability applied after projection.

Shape

Inputs:

• input: (*, input_size)

• residual: (*, output_size)

Output:

• (*, output_size)

Examples

in_size <- 4L
out_size <- 3L
model <- proj_add_norm(input_size = in_size, output_size = out_size)
input <- torch::torch_randn(in_size)
residual <- torch::torch_randn(out_size)
model(input, residual)

macmillancontentscience/torchtransformers documentation built on Aug. 6, 2023, 5:35 a.m.