decoder_byte_level: Byte level decoder
In tok: Fast Text Tokenization
decoder_byte_level R Documentation
Byte level decoder
Description
This decoder is to be used with the pre_tokenizer_byte_level.
Super class
tok_decoder -> tok_decoder_byte_level
Methods
Public methods
tok_decoder_byte_level$new()
Initializes a byte level decoder
Usage
tok_decoder_byte_level$new()
tok_decoder_byte_level$clone()
The objects of this class are cloneable with this method.
Usage
tok_decoder_byte_level$clone(deep = FALSE)
Arguments
deepWhether to make a deep clone.
See Also
Other decoders:
tok_decoder
tok documentation built on June 21, 2026, 5:07 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
| decoder_byte_level | R Documentation |
Byte level decoder
Description
This decoder is to be used with the pre_tokenizer_byte_level.
Super class
tok_decoder -> tok_decoder_byte_level
Methods
Public methods
tok_decoder_byte_level$new()
Initializes a byte level decoder
Usage
tok_decoder_byte_level$new()
tok_decoder_byte_level$clone()
The objects of this class are cloneable with this method.
Usage
tok_decoder_byte_level$clone(deep = FALSE)
Arguments
deepWhether to make a deep clone.
See Also
Other decoders:
tok_decoder
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.