LayerNormalization: LayerNormalization

In Microsoft/CNTK-R: R Interface to the Microsoft Cognitive Toolkit (CNTK)

Description

Layer factory function to create a function that implements layer normalization.

Usage

LayerNormalization(initial_scale = 1, initial_bias = 0, epsilon = 1e-05,
  name = "")

Arguments

epsilon	(float, default 0.00001) - added to avoid division by 0
name	string (optional) the name of the Function instance in the network
init	(scalar or matrix or initializer, defaults to init_glorot_uniform()) – initial value of weights Wial_scale
init_bias	(scalar or matrix or initializer, defaults to 0) – initial value of weights b

Details

Layer normalization applies this formula to every input element (element-wise): y = (x - mean(x)) / (stddev(x) + epsilon) * scale + bias where scale and bias are learned scalar parameters.

Microsoft/CNTK-R documentation built on May 28, 2019, 1:52 p.m.