tok_model: Generic class for tokenization models

In tok: Fast Text Tokenization

Generic class for tokenization models

Description

Generic class for tokenization models

Public fields

.model

stores the pointer to the model. internal

Methods

Public methods

• tok_model$new()

• tok_model$clone()

---

tok_model$new()

Initializes a genric abstract tokenizer model

Usage

tok_model$new(model)

Arguments

model

Pointer to a tokenization model

---

tok_model$clone()

The objects of this class are cloneable with this method.

Usage

tok_model$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

See Also

Other model: model_bpe, model_unigram, model_wordpiece

tok documentation built on June 21, 2026, 5:07 p.m.