Files in wordpiece
R Implementation of Wordpiece Tokenization
MD5
NEWS.md
README.md
NAMESPACE
DESCRIPTION
R/tokenization.R
R/vocab.R
R/zzz.R
R/imports.R
R/utils.R
inst/doc/basic_usage.R
inst/doc/basic_usage.Rmd
inst/doc/basic_usage.html
inst/extdata/tiny_vocab.txt
inst/extdata/tiny_vocab_cased.txt
build/vignette.rds
tests/testthat.R
tests/testthat/test-vocab.R
tests/testthat/badvocab_spaces.txt
tests/testthat/vocab.txt
tests/testthat/test-tokenization.R
tests/testthat/empty_vocab.txt
tests/testthat/badvocab_dupes.txt
tests/testthat/_snaps/vocab.md
vignettes/basic_usage.Rmd
man/dot-wp_tokenize_single_string.Rd
man/dot-validate_wordpiece_vocabulary.Rd
man/reexports.Rd
man/load_or_retrieve_vocab.Rd
man/dot-new_wordpiece_vocabulary.Rd
man/dot-process_wp_vocab.Rd
man/dot-wp_tokenize_word.Rd
man/dot-process_vocab.Rd
man/set_wordpiece_cache_dir.Rd
man/load_vocab.Rd
man/dot-infer_case_from_vocab.Rd
man/prepare_vocab.Rd
man/wordpiece_tokenize.Rd
man/wordpiece_cache_dir.Rd
man/dot-get_casedness.Rd
