Custom Activation Function"

In kindling: Higher-Level Interface of 'torch' Package to Auto-Train Neural Networks

knitr::opts_chunk$set(
    eval = identical(Sys.getenv("NOT_CRAN"), "true"),
    fig.width = 7,
    fig.height = 5,
    warning = FALSE,
    message = FALSE
)
library(kindling)

Rationale

The biggest strength of {kindling} when modelling neural networks is its versatility — it inherits {torch}'s versatility while being human friendly, including the ability to apply custom optimizer functions, loss functions, and per-layer activation functions. Learn more: https://kindling.joshuamarie.com/articles/special-cases.

With act_funs(), you are not limited to the activation functions available in {torch}'s namespace. Use new_act_fn() to wrap any compatible function into a validated custom activation. This feature, however, is only available on version 0.3.0 and above.

Function to use

To do this, use new_act_fn(). It takes a user-supplied function, validates it against a small dummy tensor at definition time (a dry-run probe), and wraps it in a call-time type guard. This means errors surface early — before your model ever starts training.

The function you supply must:

• Accept at least one argument (the input tensor).
• Return a torch_tensor.

Basic Usage

Currently, nnf_tanh doesn't exist in the {torch} namespace, so tanh is not a valid argument to act_funs(). With new_act_fn(), you can wrap torch::torch_tanh() to make it usable.

Here's a basic example that wraps torch::torch_tanh() as a custom activation:

hyper_tan = new_act_fn(\(x) torch::torch_tanh(x))

You can also pass it directly into act_funs(), just like any built-in activation:

act_funs(relu, elu, new_act_fn(\(x) torch::torch_tanh(x)))

Using Custom Activations in a Model

Naturally, functions for modelling, like ffnn(), accept act_funs() into the activations argument. Again, you can pass a custom activation function within new_act_fn(), then pass it through act_funs().

Here's a basic example:

model = ffnn(
    Sepal.Length ~ .,
    data = iris[, 1:4],
    hidden_neurons = c(64, 32, 16),
    activations = act_funs(
        relu,
        silu,
        new_act_fn(\(x) torch::torch_tanh(x))
    ),
    epochs = 50
)
model

Each element of act_funs() corresponds to one hidden layer, in order. Here, the first hidden layer uses ReLU, the second uses SiLU (Swish), and the third uses Tanh.

You can also use a single custom activation recycled across all layers:

ffnn(
    Sepal.Length ~ .,
    data = iris[, 1:4],
    hidden_neurons = c(64, 32),
    activations = act_funs(new_act_fn(\(x) torch::torch_tanh(x))),
    epochs = 50
)

Skipping the Dry-Run Probe

By default, new_act_fn() runs a quick dry-run with a small dummy tensor to validate your function before training. You can disable this with probe = FALSE, though this is generally not recommended:

my_act = new_act_fn(\(x) torch::torch_tanh(x), probe = FALSE)

Naming Your Custom Activation

You can provide a human-readable name via .name, which is used in print output and diagnostics:

my_act = new_act_fn(\(x) torch::torch_tanh(x), .name = "my_tanh")

Here's a simple application:

ffnn(
    Sepal.Length ~ .,
    data = iris[, 1:4],
    hidden_neurons = c(64, 32),
    activations = act_funs(
        relu, 
        new_act_fn(\(x) torch::torch_tanh(x), .name = "hyper_tanh")
    ),
    epochs = 50
)

Error Handling

new_act_fn() is designed to fail loudly and early. Common errors include:

1. Function returns a non-tensor. This will error at definition time:

   r new_act_fn(\(x) as.numeric(x))

2. Function accepts no arguments. This will error immediately:

   r new_act_fn(function() torch::torch_zeros(2))

These checks ensure your model's architecture is valid before any data ever flows through it.

Summary

| Feature | Details | |---|---| | Wraps any R function | Must accept a tensor, return a tensor | | Dry-run probe | Validates at definition time (probe = TRUE by default) | | Call-time guard | Type-checks output on every forward pass | | Compatible with act_funs() | Use alongside built-in activations freely | | Closures supported | Parametric activations work naturally |

kindling documentation built on March 3, 2026, 9:07 a.m.