tok-package: tok: Fast Text Tokenization
In tok: Fast Text Tokenization
tok-package R Documentation
tok: Fast Text Tokenization
Description
Interfaces with the 'Hugging Face' tokenizers library to provide implementations of today's most used tokenizers such as the 'Byte-Pair Encoding' algorithm https://huggingface.co/docs/tokenizers/index. It's extremely fast for both training new vocabularies and tokenizing texts.
Author(s)
Maintainer: Daniel Falbel daniel@posit.co
Other contributors:
Posit [copyright holder]
See Also
Useful links:
tok documentation built on Sept. 11, 2024, 5:21 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
| tok-package | R Documentation |
tok: Fast Text Tokenization
Description
Interfaces with the 'Hugging Face' tokenizers library to provide implementations of today's most used tokenizers such as the 'Byte-Pair Encoding' algorithm https://huggingface.co/docs/tokenizers/index. It's extremely fast for both training new vocabularies and tokenizing texts.
Author(s)
Maintainer: Daniel Falbel daniel@posit.co
Other contributors:
Posit [copyright holder]
See Also
Useful links:
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.