processor_byte_level: Byte Level post processor

In tok: Fast Text Tokenization

Byte Level post processor

Description

This post-processor takes care of trimming the offsets. By default, the ByteLevel BPE might include whitespaces in the produced tokens. If you don’t want the offsets to include these whitespaces, then this PostProcessor must be used.

Super class

tok_processor -> tok_processor_byte_level

Methods

Public methods

• tok_processor_byte_level$new()

• tok_processor_byte_level$clone()

---

tok_processor_byte_level$new()

Initializes the byte level post processor

Usage

tok_processor_byte_level$new(trim_offsets = TRUE)

Arguments

trim_offsets

Whether to trim the whitespaces from the produced offsets.

---

tok_processor_byte_level$clone()

The objects of this class are cloneable with this method.

Usage

tok_processor_byte_level$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

See Also

Other processors: tok_processor

tok documentation built on June 21, 2026, 5:07 p.m.