tok_normalizer: Generic class for normalizers

In tok: Fast Text Tokenization

Generic class for normalizers

Description

Generic class for normalizers

Generic class for normalizers

Public fields

.normalizer

Internal pointer to normalizer object

Methods

Public methods

• tok_normalizer$new()

• tok_normalizer$clone()

---

Method new()

Initializes a tokenizer

Usage

tok_normalizer$new(normalizer)

Arguments

normalizer

a raw pointer to a tokenizer

---

Method clone()

The objects of this class are cloneable with this method.

Usage

tok_normalizer$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

See Also

Other normalizers: normalizer_nfc, normalizer_nfkc

tok documentation built on Sept. 11, 2024, 5:21 p.m.