add_timing_signal_1d: Add timing signal to a tensor.

Description Usage Arguments

View source: R/positional-encodings.R

Description

Adds a bunch of sinusoids of different frequencies to a Tensor. Each channel of the input Tensor is incremented by a sinusoid of a different frequency and phase. This allows attention to learn to use absolute and relative positions. Timing signals should be added to some precursors of both the query and the memory inputs to attention. The use of relative position is possible because sin(x+y) and cos(x+y) can be experessed in terms of y, sin(x) and cos(x). In particular, we use a geometric sequence of timescales starting with min_timescale and ending with max_timescale. The number of different timescales is equal to channels / 2. For each timescale, we generate the two sinusoidal signals sin(timestep/timescale) and cos(timestep/timescale). All of these sinusoids are concatenated in the channels dimension.

Usage

1
2
3
4
5
6
add_timing_signal_1d(
  x,
  min_timescale = 1,
  max_timescale = 10000,
  start_index = 0L
)

Arguments

x:

a Tensor with shape [batch, length, channels]

min_timescale:

a float

max_timescale:

a float

start_index:

index of first position


ifrit98/transformR documentation built on Nov. 26, 2019, 2:14 a.m.