Files in macmillancontentscience/wordpiece
R Implementation of Wordpiece Tokenization
.Rbuildignore
.github/.gitignore
.github/CONTRIBUTING.md
.gitignore
CODE_OF_CONDUCT.md
DESCRIPTION
LICENSE.md
NAMESPACE
NEWS.md
R/imports.R
R/tokenization.R
R/utils.R
R/vocab.R
R/zzz.R
README.Rmd
README.md
cran-comments.md
inst/extdata/tiny_vocab.txt
inst/extdata/tiny_vocab_cased.txt
man/dot-get_casedness.Rd
man/dot-infer_case_from_vocab.Rd
man/dot-new_wordpiece_vocabulary.Rd
man/dot-process_vocab.Rd
man/dot-process_wp_vocab.Rd
man/dot-validate_wordpiece_vocabulary.Rd
man/dot-wp_tokenize_single_string.Rd
man/dot-wp_tokenize_word.Rd
man/load_or_retrieve_vocab.Rd
man/load_vocab.Rd
man/prepare_vocab.Rd
man/reexports.Rd
man/set_wordpiece_cache_dir.Rd
man/wordpiece_cache_dir.Rd
man/wordpiece_tokenize.Rd
tests/testthat.R
tests/testthat/_snaps/vocab.md
tests/testthat/badvocab_dupes.txt
tests/testthat/badvocab_spaces.txt
tests/testthat/empty_vocab.txt
tests/testthat/test-tokenization.R
tests/testthat/test-vocab.R
tests/testthat/vocab.txt
vignettes/.gitignore
vignettes/basic_usage.Rmd
wordpiece.Rproj
