get_token_count: Returns the number of tokens in a text
In rtiktoken: A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models
View source: R/get_token_count.R
get_token_count R Documentation
Returns the number of tokens in a text
Description
Returns the number of tokens in a text
Usage
get_token_count(text, model)
Arguments
text
a character string to encode to tokens, can be a vector
model
a model to use for tokenization, either a model name, e.g., gpt-4o
or a tokenizer, e.g., o200k_base.
See also available tokenizers.
Value
the number of tokens in the text, vector of integers
See Also
model_to_tokenizer(), get_tokens()
Examples
get_token_count("Hello World", "gpt-4o")
rtiktoken documentation built on April 15, 2025, 1:35 a.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
View source: R/get_token_count.R
| get_token_count | R Documentation |
Returns the number of tokens in a text
Description
Returns the number of tokens in a text
Usage
get_token_count(text, model)
Arguments
text |
a character string to encode to tokens, can be a vector |
model |
a model to use for tokenization, either a model name, e.g., |
Value
the number of tokens in the text, vector of integers
See Also
model_to_tokenizer(), get_tokens()
Examples
get_token_count("Hello World", "gpt-4o")
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.