get_tokens: Converts text to tokens
In rtiktoken: A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models
get_tokens R Documentation
Converts text to tokens
Description
Converts text to tokens
Usage
get_tokens(text, model)
Arguments
text
a character string to encode to tokens, can be a vector
model
a model to use for tokenization, either a model name, e.g., gpt-4o
or a tokenizer, e.g., o200k_base.
See also available tokenizers.
Value
a vector of tokens for the given text as integer
See Also
model_to_tokenizer(), decode_tokens()
Examples
get_tokens("Hello World", "gpt-4o")
get_tokens("Hello World", "o200k_base")
rtiktoken documentation built on April 15, 2025, 1:35 a.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
| get_tokens | R Documentation |
Converts text to tokens
Description
Converts text to tokens
Usage
get_tokens(text, model)
Arguments
text |
a character string to encode to tokens, can be a vector |
model |
a model to use for tokenization, either a model name, e.g., |
Value
a vector of tokens for the given text as integer
See Also
model_to_tokenizer(), decode_tokens()
Examples
get_tokens("Hello World", "gpt-4o")
get_tokens("Hello World", "o200k_base")
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.