mlr_pipeops_nn_ft_cls: CLS Token for FT-Transformer

In mlr-org/mlr3torch: Deep Learning with 'mlr3'

CLS Token for FT-Transformer

Description

Concatenates a CLS token to the input as the last feature. The input shape is expected to be ⁠(batch, n_features, d_token)⁠ and the output shape is ⁠(batch, n_features + 1, d_token)⁠.

This is used in the FT-Transformer.

nn_module

Calls nn_ft_cls() when trained.

State

The state is the value calculated by the public method ⁠$shapes_out()⁠.

Super classes

mlr3pipelines::PipeOp -> mlr3torch::PipeOpTorch -> PipeOpTorchFTCLS

Methods

Public methods

• PipeOpTorchFTCLS$new()

• PipeOpTorchFTCLS$clone()

Inherited methods

• mlr3pipelines::PipeOp$help()
• mlr3pipelines::PipeOp$predict()
• mlr3pipelines::PipeOp$print()
• mlr3pipelines::PipeOp$train()
• mlr3torch::PipeOpTorch$shapes_out()

---

Method new()

Creates a new instance of this R6 class.

Usage

PipeOpTorchFTCLS$new(id = "nn_ft_cls", param_vals = list())

Arguments

id

(character(1))
Identifier of the resulting object.

param_vals

(list())
List of hyperparameter settings, overwriting the hyperparameter settings that would otherwise be set during construction.

---

Method clone()

The objects of this class are cloneable with this method.

Usage

PipeOpTorchFTCLS$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

See Also

Other PipeOps: mlr_pipeops_nn_adaptive_avg_pool1d, mlr_pipeops_nn_adaptive_avg_pool2d, mlr_pipeops_nn_adaptive_avg_pool3d, mlr_pipeops_nn_avg_pool1d, mlr_pipeops_nn_avg_pool2d, mlr_pipeops_nn_avg_pool3d, mlr_pipeops_nn_batch_norm1d, mlr_pipeops_nn_batch_norm2d, mlr_pipeops_nn_batch_norm3d, mlr_pipeops_nn_block, mlr_pipeops_nn_celu, mlr_pipeops_nn_conv1d, mlr_pipeops_nn_conv2d, mlr_pipeops_nn_conv3d, mlr_pipeops_nn_conv_transpose1d, mlr_pipeops_nn_conv_transpose2d, mlr_pipeops_nn_conv_transpose3d, mlr_pipeops_nn_dropout, mlr_pipeops_nn_elu, mlr_pipeops_nn_flatten, mlr_pipeops_nn_geglu, mlr_pipeops_nn_gelu, mlr_pipeops_nn_glu, mlr_pipeops_nn_hardshrink, mlr_pipeops_nn_hardsigmoid, mlr_pipeops_nn_hardtanh, mlr_pipeops_nn_head, mlr_pipeops_nn_identity, mlr_pipeops_nn_layer_norm, mlr_pipeops_nn_leaky_relu, mlr_pipeops_nn_linear, mlr_pipeops_nn_log_sigmoid, mlr_pipeops_nn_max_pool1d, mlr_pipeops_nn_max_pool2d, mlr_pipeops_nn_max_pool3d, mlr_pipeops_nn_merge, mlr_pipeops_nn_merge_cat, mlr_pipeops_nn_merge_prod, mlr_pipeops_nn_merge_sum, mlr_pipeops_nn_prelu, mlr_pipeops_nn_reglu, mlr_pipeops_nn_relu, mlr_pipeops_nn_relu6, mlr_pipeops_nn_reshape, mlr_pipeops_nn_rrelu, mlr_pipeops_nn_selu, mlr_pipeops_nn_sigmoid, mlr_pipeops_nn_softmax, mlr_pipeops_nn_softplus, mlr_pipeops_nn_softshrink, mlr_pipeops_nn_softsign, mlr_pipeops_nn_squeeze, mlr_pipeops_nn_tanh, mlr_pipeops_nn_tanhshrink, mlr_pipeops_nn_threshold, mlr_pipeops_nn_tokenizer_categ, mlr_pipeops_nn_tokenizer_num, mlr_pipeops_nn_unsqueeze, mlr_pipeops_torch_ingress, mlr_pipeops_torch_ingress_categ, mlr_pipeops_torch_ingress_ltnsr, mlr_pipeops_torch_ingress_num, mlr_pipeops_torch_loss, mlr_pipeops_torch_model, mlr_pipeops_torch_model_classif, mlr_pipeops_torch_model_regr

Examples

# Construct the PipeOp
pipeop = po("nn_ft_cls")
pipeop
# The available parameters
pipeop$param_set

mlr-org/mlr3torch documentation built on April 17, 2025, 8:22 p.m.