Man pages for rtiktoken
A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models
| decode_tokens | Decodes tokens back to text |
| get_token_count | Returns the number of tokens in a text |
| get_tokens | Converts text to tokens |
| model_to_tokenizer | Gets the name of the tokenizer used by a model |
