Files in rtiktoken
A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models
configure
MD5
NEWS.md
README.md
NAMESPACE
DESCRIPTION
LICENSE
configure.win
R/extendr-wrappers.R
R/model_to_tokenizer.R
R/get_tokens.R
R/decode.R
R/get_token_count.R
src/entrypoint.c
src/rtiktoken-win.def
src/Makevars.win.in
src/Makevars.in
src/rust/Cargo.toml
src/rust/vendor.tar.xz
src/rust/vendor-packages.sh
src/rust/Cargo.lock
src/rust/vendor-config.toml
src/rust/src/lib.rs
inst/AUTHORS
tests/testthat.R
tests/testthat/test-tokens.R
tools/config.R
tools/msrv.R
man/get_token_count.Rd
man/get_tokens.Rd
man/decode_tokens.Rd
man/model_to_tokenizer.Rd
