tok_model: Generic class for tokenization models

In tok: Fast Text Tokenization

Generic class for tokenization models

Description

Generic class for tokenization models

Generic class for tokenization models

Public fields

.model

stores the pointer to the model. internal

Methods

Public methods

• tok_model$new()

• tok_model$clone()

---

Method new()

Initializes a genric abstract tokenizer model

Usage

tok_model$new(model)

Arguments

model

Pointer to a tokenization model

---

Method clone()

The objects of this class are cloneable with this method.

Usage

tok_model$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

See Also

Other model: model_bpe, model_unigram, model_wordpiece

tok documentation built on Sept. 11, 2024, 5:21 p.m.