R/pkg.R
In tokenizers.bpe: Byte Pair Encoding Text Tokenization

#' @importFrom Rcpp evalCpp
#' @useDynLib tokenizers.bpe
NULL


.loaded_models <- new.env()

Any scripts or data that you put into this service are public.

tokenizers.bpe documentation built on Sept. 16, 2023, 1:06 a.m.

Note that we can't provide technical support on individual packages. You should contact the package authors for that.

Try the tokenizers.bpe package in your browser