decoder_byte_level: Byte level decoder

In tok: Fast Text Tokenization

Byte level decoder

Description

Byte level decoder

Byte level decoder

Details

This decoder is to be used with the pre_tokenizer_byte_level.

Super class

tok::tok_decoder -> tok_decoder_byte_level

Methods

Public methods

• decoder_byte_level$new()

• decoder_byte_level$clone()

---

Method new()

Initializes a byte level decoder

Usage

decoder_byte_level$new()

---

Method clone()

The objects of this class are cloneable with this method.

Usage

decoder_byte_level$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

See Also

Other decoders: tok_decoder

tok documentation built on Sept. 11, 2024, 5:21 p.m.