tok_processor: Generic class for processors

In tok: Fast Text Tokenization

Generic class for processors

Description

Generic class for processors

Generic class for processors

Public fields

.processor

Internal pointer to processor object

Methods

Public methods

• tok_processor$new()

• tok_processor$clone()

---

Method new()

Initializes a tokenizer

Usage

tok_processor$new(processor)

Arguments

processor

a raw pointer to a processor

---

Method clone()

The objects of this class are cloneable with this method.

Usage

tok_processor$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

See Also

Other processors: processor_byte_level

tok documentation built on Sept. 11, 2024, 5:21 p.m.