decode_tokens: Decodes tokens back to text

In rtiktoken: A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models

Decodes tokens back to text

Description

Decodes tokens back to text

Usage

decode_tokens(tokens, model)

Arguments

tokens	a vector of tokens to decode, or a list of tokens
model	a model to use for tokenization, either a model name, e.g., ⁠gpt-4o⁠ or a tokenizer, e.g., o200k_base. See also available tokenizers.

Value

a character string of the decoded tokens or a vector or strings

See Also

model_to_tokenizer(), get_tokens()

Examples

tokens <- get_tokens("Hello World", "gpt-4o")
tokens
decode_tokens(tokens, "gpt-4o")

tokens <- get_tokens(c("Hello World", "Alice Bob Charlie"), "gpt-4o")
tokens
decode_tokens(tokens, "gpt-4o")

rtiktoken documentation built on April 15, 2025, 1:35 a.m.