activation: Activation function

In dnn: Deep Neural Network Tools for Probability and Statistic Models

Activation function

Description

Different type of activation functions and the corresponding derivatives

Usage

  sigmoid(x)
  elu(x)
  relu(x)
  lrelu(x)
  idu(x)
  dsigmoid(y)
  delu(y)
  drelu(y)
  dlrelu(y)
  dtanh(y)   #activation function tanh(x) is already available in R

Arguments

x	input of the activation function
y	input of the derivative of the activation function

Details

Each function returns either the activation function (e.g. sigmoid, relu) or its derivative (e.g. dsigmoid, drelu).

Value

An activation function is applied to x and returns a matrix the same size as x. The detail formula for each activation function is:

sigmoid	return 1/(1+exp(-x))
elu	return x for x>0 and exp(x)-1 for x<0
relu	return x for x>0 and 0 for x<0
lrelu	return x for x>0 and 0.1*x for x<0
tanh	return tanh(x)
idu	return (x)

Author(s)

Bingshu E. Chen

See Also

bwdNN, fwdNN, dNNmodel, optimizerSGD, optimizerNAG

Examples

  # Specify a dnn nodel with user define activation function in layer 2.
  softmax  = function(x) {log(1+exp(x))}    # y = log(1+exp(x))
  dsoftmax = function(y) {sigmoid(y)}       # x = exp(y)/(1+exp(y))
  model = dNNmodel(units=c(8, 6, 1), activation= c('relu', 'softmax', 'sigmoid'), 
          input_shape = c(3))
  print(model)

dnn documentation built on May 29, 2024, 1:48 a.m.