model_to_tokenizer: Gets the name of the tokenizer used by a model
In rtiktoken: A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models
View source: R/model_to_tokenizer.R
model_to_tokenizer R Documentation
Gets the name of the tokenizer used by a model
Description
Gets the name of the tokenizer used by a model
Usage
model_to_tokenizer(model)
Arguments
model
the model to use, e.g., gpt-4o
Value
the tokenizer used by the model
Examples
model_to_tokenizer("gpt-4o")
model_to_tokenizer("gpt-4-1106-preview")
model_to_tokenizer("text-davinci-002")
model_to_tokenizer("text-embedding-ada-002")
model_to_tokenizer("text-embedding-3-small")
rtiktoken documentation built on April 15, 2025, 1:35 a.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
View source: R/model_to_tokenizer.R
| model_to_tokenizer | R Documentation |
Gets the name of the tokenizer used by a model
Description
Gets the name of the tokenizer used by a model
Usage
model_to_tokenizer(model)
Arguments
model |
the model to use, e.g., |
Value
the tokenizer used by the model
Examples
model_to_tokenizer("gpt-4o")
model_to_tokenizer("gpt-4-1106-preview")
model_to_tokenizer("text-davinci-002")
model_to_tokenizer("text-embedding-ada-002")
model_to_tokenizer("text-embedding-3-small")
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.