processor_byte_level: Byte Level post processor

In tok: Fast Text Tokenization

Byte Level post processor

Description

Byte Level post processor

Byte Level post processor

Details

This post-processor takes care of trimming the offsets. By default, the ByteLevel BPE might include whitespaces in the produced tokens. If you don’t want the offsets to include these whitespaces, then this PostProcessor must be used.

Super class

tok::tok_processor -> tok_processor_byte_level

Methods

Public methods

• processor_byte_level$new()

• processor_byte_level$clone()

---

Method new()

Initializes the byte level post processor

Usage

processor_byte_level$new(trim_offsets = TRUE)

Arguments

trim_offsets

Whether to trim the whitespaces from the produced offsets.

---

Method clone()

The objects of this class are cloneable with this method.

Usage

processor_byte_level$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

See Also

Other processors: tok_processor

tok documentation built on Sept. 11, 2024, 5:21 p.m.