num_tokens_file: Compute total number of tokens in a text file
In TheOpenAIR: Integrate 'OpenAI' Large Language Models into Your 'R' Workflows
View source: R/num_tokens_file.R
num_tokens_file R Documentation
Compute total number of tokens in a text file
Description
The function batch-wise computes the total number of tokens in a text file.
The function returns a numeric value indicating the total
number of tokens in the file. The function can be used on very large text files.
Usage
num_tokens_file(filename, batch_size = 1000, encoding = "cl100k_base")
Arguments
filename
character string indicating the name of the text file to read in
batch_size
integer indicating the number of lines to read in per batch (default is 1000)
encoding
character string indicating the encoding to use (default is "cl100k_base")
Value
a numeric value indicating the total number of tokens in the text file
TheOpenAIR documentation built on April 27, 2023, 5:10 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
View source: R/num_tokens_file.R
| num_tokens_file | R Documentation |
Compute total number of tokens in a text file
Description
The function batch-wise computes the total number of tokens in a text file. The function returns a numeric value indicating the total number of tokens in the file. The function can be used on very large text files.
Usage
num_tokens_file(filename, batch_size = 1000, encoding = "cl100k_base")
Arguments
filename |
character string indicating the name of the text file to read in |
batch_size |
integer indicating the number of lines to read in per batch (default is 1000) |
encoding |
character string indicating the encoding to use (default is "cl100k_base") |
Value
a numeric value indicating the total number of tokens in the text file
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.