num_tokens: Get number of tokens in a string using OpenAI's tiktoken...
In TheOpenAIR: Integrate 'OpenAI' Large Language Models into Your 'R' Workflows
num_tokens R Documentation
Get number of tokens in a string using OpenAI's tiktoken library
Description
This function uses the num_tokens_from_string function provided in OpenAI's
tiktoken python library to get the number of tokens in a string.
Usage
num_tokens(text, encoding = "cl100k_base")
Arguments
text
a character string to count the number of tokens from
encoding
a character string that specifies how text is converted into tokens. The default is "cl100k_base" (for ChatGPT models; see https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb for details)
Value
an integer indicating the number of tokens in the input string
References
https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
TheOpenAIR documentation built on April 27, 2023, 5:10 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
| num_tokens | R Documentation |
Get number of tokens in a string using OpenAI's tiktoken library
Description
This function uses the num_tokens_from_string function provided in OpenAI's tiktoken python library to get the number of tokens in a string.
Usage
num_tokens(text, encoding = "cl100k_base")
Arguments
text |
a character string to count the number of tokens from |
encoding |
a character string that specifies how text is converted into tokens. The default is "cl100k_base" (for ChatGPT models; see https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb for details) |
Value
an integer indicating the number of tokens in the input string
References
https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.