magicTokenizer: magicTokenizer
In MacBehaviour: Behavioural Studies of Large Language Models
magicTokenizer R Documentation
magicTokenizer
Description
This function provides the number of tokens in a specified text, acting as a wrapper for an internal tokenizer function.
Usage
magicTokenizer(text)
Arguments
text
A character string: the text for which the number of tokens is required.
Value
Returns the number of tokens in the provided text.
MacBehaviour documentation built on Oct. 21, 2024, 1:07 a.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
| magicTokenizer | R Documentation |
magicTokenizer
Description
This function provides the number of tokens in a specified text, acting as a wrapper for an internal tokenizer function.
Usage
magicTokenizer(text)
Arguments
text |
A character string: the text for which the number of tokens is required. |
Value
Returns the number of tokens in the provided text.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.