Files in tokenizers

Fast, Consistent Tokenization of Natural Language Text

MD5

NEWS.md README.md

NAMESPACE

DESCRIPTION

LICENSE

R/RcppExports.R R/data-docs.R R/stem-tokenizers.R R/coercion.R R/ptb-tokenizer.R R/tokenizers-package.r R/ngram-tokenizers.R R/chunk-text.R R/basic-tokenizers.R R/wordcount.R R/utils.R R/character-shingles-tokenizers.R

src/RcppExports.cpp

src/shingle_ngrams.cpp

src/skip_ngrams.cpp

inst/CITATION

inst/doc/tif-and-tokenizers.html

inst/doc/introduction-to-tokenizers.html

inst/doc/tif-and-tokenizers.Rmd inst/doc/introduction-to-tokenizers.R inst/doc/introduction-to-tokenizers.Rmd inst/doc/tif-and-tokenizers.R

build/vignette.rds

tests/testthat.R tests/testthat/test-utils.R tests/testthat/test-chunking.R

tests/testthat/moby-ch1.txt

tests/testthat/helper-data.R tests/testthat/test-ngrams.R tests/testthat/test-ptb.R tests/testthat/test-shingles.R

tests/testthat/moby-ch2.txt

tests/testthat/test-basic.R

tests/testthat/moby-ch3.txt

tests/testthat/test-encoding.R tests/testthat/test-tif.R tests/testthat/test-wordcount.R tests/testthat/test-stem.R vignettes/tif-and-tokenizers.Rmd vignettes/introduction-to-tokenizers.Rmd

data/mobydick.rda

man/tokenizers.Rd man/chunk_text.Rd man/ngram-tokenizers.Rd man/mobydick.Rd man/shingle-tokenizers.Rd man/ptb-tokenizer.Rd man/word-counting.Rd man/stem-tokenizers.Rd man/basic-tokenizers.Rd

tokenizers documentation built on Dec. 28, 2022, 2:34 a.m.