Nothing
tok: Fast Text Tokenization
Interfaces with the 'Hugging Face' tokenizers library to provide implementations of today's most used tokenizers such as the 'Byte-Pair Encoding' algorithm <https://huggingface.co/docs/tokenizers/index>. It's extremely fast for both training new vocabularies and tokenizing texts.
Getting started
Browse package contents
Package details |
|
|---|---|
| Author | Tomasz Kalinowski [ctb, cre], Daniel Falbel [aut], Regouby Christophe [ctb], Posit [cph] |
| Maintainer | Tomasz Kalinowski <tomasz@posit.co> |
| License | MIT + file LICENSE |
| Version | 0.2.3 |
| URL | https://github.com/mlverse/tok |
| Package repository | View on CRAN |
| Installation |
Install the latest version of this package by entering the following in R:
|
Try the tok package in your browser
Any scripts or data that you put into this service are public.
