ntokens: The number of tokens in a string or vector of strings

In pangoling: Access to Large Language Model Predictions

The number of tokens in a string or vector of strings

Description

The number of tokens in a string or vector of strings

Usage

ntokens(
  x,
  model = getOption("pangoling.causal.default"),
  add_special_tokens = NULL,
  config_tokenizer = NULL
)

Arguments

x	character input
model	Name of a pre-trained model or folder. One should be able to use models based on "gpt2". See hugging face website.
add_special_tokens	Whether to include special tokens. It has the same default as the AutoTokenizer method in Python.
config_tokenizer	List with other arguments that control how the tokenizer from Hugging Face is accessed.

Value

The number of tokens in a string or vector of words.

See Also

Other token-related functions: tokenize_lst(), transformer_vocab()

Examples

ntokens(x = c("The apple doesn't fall far from the tree."), model = "gpt2")

pangoling documentation built on April 11, 2025, 6:16 p.m.