bias_act: Fused bias and activation function.

View source: R/bias_act.R

bias_actR Documentation

Fused bias and activation function.

Description

Adds bias b to activation tensor x, evaluates activation function act, and scales the result by gain. Each of the steps is optional. In most cases, the fused op is considerably more efficient than performing the same calculation using standard PyTorch ops. It supports first and second order gradients, but not third order gradients.

Usage

bias_act(
  x,
  b = NULL,
  dim = 2,
  act = "linear",
  alpha = NULL,
  gain = NULL,
  clamp = NULL,
  impl = if (cuda_is_available() & x$device$type == "cuda") "cuda" else "ref"
)

Arguments

x

Input activation tensor. Can be of any shape.

b

Bias vector, or NULL to disable. Must be a 1D tensor of the same type as x. The shape must be known, and it must match the dimension of x corresponding to dim.

dim

The dimension in x corresponding to the elements of b. The value of dim is ignored if b is not specified.

act

Name of the activation function to evaluate, or "linear" to disable. Can be e.g. "relu", "lrelu", "tanh", "sigmoid", "swish", etc. See details for a full list. NULL is not allowed.

alpha

Shape parameter for the activation function, or NULL to use the default.

gain

Scaling factor for the output tensor, or NULL to use default. See details for the default scaling of each activation function. If unsure, consider specifying 1.

clamp

Clamp the output values to c(-clamp, +clamp), or NULL to disable the clamping (default).

impl

Name of the implementation to use. Can be "ref" or "cuda".

Value

torch_tensor of the same shape and datatype as x.

Copyright

Note that this function used code from the StyleGAN3 project which is copyright of Nvidia 2021, and is redistributed in the torch package under its original license which can be found here: https://github.com/NVlabs/stylegan3/blob/main/LICENSE.txt. Note that under the license use is restricted to non-commercial purposes. If you use this function, please make sure your use is acceptable under the license linked above.#'


rdinnager/styleganr documentation built on Nov. 9, 2022, 6:09 a.m.