Nothing
tok: Fast Text Tokenization
Interfaces with the 'Hugging Face' tokenizers library to provide implementations of today's most used tokenizers such as the 'Byte-Pair Encoding' algorithm <https://huggingface.co/docs/tokenizers/index>. It's extremely fast for both training new vocabularies and tokenizing texts.
Getting started
Browse package contents
Package details |
|
|---|---|
| Author | Daniel Falbel [aut, cre], Posit [cph] |
| Maintainer | Daniel Falbel <daniel@posit.co> |
| License | MIT + file LICENSE |
| Version | 0.1.4 |
| URL | https://github.com/mlverse/tok |
| Package repository | View on CRAN |
| Installation |
Install the latest version of this package by entering the following in R:
|
Try the tok package in your browser
Any scripts or data that you put into this service are public.
