model_unigram: An implementation of the Unigram algorithm

In tok: Fast Text Tokenization

An implementation of the Unigram algorithm

Description

An implementation of the Unigram algorithm

Super class

tok_model -> tok_model_unigram

Methods

Public methods

• tok_model_unigram$new()

• tok_model_unigram$clone()

---

tok_model_unigram$new()

Constructor for Unigram Model

Usage

tok_model_unigram$new(vocab = NULL, unk_id = NULL, byte_fallback = FALSE)

Arguments

vocab

A dictionary of string keys and their corresponding relative score. Default: NULL.

unk_id

The unknown token id to be used by the model. Default: NULL.

byte_fallback

Whether to use byte-fallback trick. Default: FALSE.

---

tok_model_unigram$clone()

The objects of this class are cloneable with this method.

Usage

tok_model_unigram$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

See Also

Other model: model_bpe, model_wordpiece, tok_model

tok documentation built on June 21, 2026, 5:07 p.m.