API for rtiktoken
A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models
| Global functions | |
|---|---|
| decode_tokens | Man page Source code |
| decode_tokens_internal | Source code |
| get_token_count | Man page Source code |
| get_tokens | Man page Source code |
| get_tokens_internal | Source code |
| model_to_tokenizer | Man page Source code |
| rs_decode_tokens | Source code |
| rs_get_token_count | Source code |
| rs_get_tokens | Source code |
| rs_model_to_tokenizer | Source code |
