Nothing
rtiktoken: A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models
A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.
Getting started
Browse package contents
Package details |
|
|---|---|
| Author | David Zimmermann-Kollenda [aut, cre], Roger Zurawicki [aut] (tiktoken-rs Rust library), Authors of the dependent Rust crates [aut] (see AUTHORS file) |
| Maintainer | David Zimmermann-Kollenda <david_j_zimmermann@hotmail.com> |
| License | MIT + file LICENSE |
| Version | 0.0.7 |
| URL | https://davzim.github.io/rtiktoken/ https://github.com/DavZim/rtiktoken/ |
| Package repository | View on CRAN |
| Installation |
Install the latest version of this package by entering the following in R:
|
Try the rtiktoken package in your browser
Any scripts or data that you put into this service are public.
