Files in tokenizers.bpe
Byte Pair Encoding Text Tokenization

MD5
NEWS.md README.md
LICENSE.note
NAMESPACE
DESCRIPTION
R/RcppExports.R R/data.R R/pkg.R R/youtokentome.R
src/Makevars
src/rcpp_youtokentome.cpp
src/RcppExports.cpp
src/parallel_hashmap/phmap_fwd_decl.h
src/parallel_hashmap/meminfo.h
src/parallel_hashmap/phmap_config.h
src/parallel_hashmap/phmap_base.h
src/parallel_hashmap/phmap.h
src/parallel_hashmap/phmap_utils.h
src/parallel_hashmap/phmap_dump.h
src/parallel_hashmap/btree.h
src/parallel_hashmap/phmap_bits.h
src/youtokentome/cpp/bpe.h
src/youtokentome/cpp/bpe.cpp
src/youtokentome/cpp/utf8.h
src/youtokentome/cpp/utf8.cpp
src/youtokentome/cpp/utils.cpp
src/youtokentome/cpp/utils.h
inst/extdata/youtokentome.bpe
data/belgium_parliament.RData
man/bpe_load_model.Rd man/belgium_parliament.Rd man/bpe_decode.Rd man/bpe_encode.Rd man/bpe.Rd
tokenizers.bpe documentation built on Sept. 16, 2023, 1:06 a.m.