Files in lmullen/tokenizers

Fast, Consistent Tokenization of Natural Language Text

.Rbuildignore

.gitignore

.travis.yml

CONDUCT.md

CRAN-SUBMISSION

DESCRIPTION

Dockerfile

LICENSE

Makefile

NAMESPACE

NEWS.md R/RcppExports.R R/basic-tokenizers.R R/character-shingles-tokenizers.R R/chunk-text.R R/coercion.R R/data-docs.R R/ngram-tokenizers.R R/ptb-tokenizer.R R/stem-tokenizers.R R/tokenizers-package.r R/utils.R R/wordcount.R README.Rmd README.md

_pkgdown.yml

appveyor.yml

cran-comments.md

data-raw/moby-dick.txt

data-raw/mobydick.R

data/mobydick.rda

docker-compose.yml

inst/CITATION

man/basic-tokenizers.Rd man/chunk_text.Rd man/mobydick.Rd man/ngram-tokenizers.Rd man/ptb-tokenizer.Rd man/shingle-tokenizers.Rd man/stem-tokenizers.Rd man/tokenizers.Rd man/word-counting.Rd

paper.bib

paper.md

src/RcppExports.cpp

src/shingle_ngrams.cpp

src/skip_ngrams.cpp

tests/testthat.R tests/testthat/helper-data.R

tests/testthat/moby-ch1.txt

tests/testthat/moby-ch2.txt

tests/testthat/moby-ch3.txt

tests/testthat/test-basic.R tests/testthat/test-chunking.R tests/testthat/test-encoding.R tests/testthat/test-ngrams.R tests/testthat/test-ptb.R tests/testthat/test-shingles.R tests/testthat/test-stem.R tests/testthat/test-tif.R tests/testthat/test-utils.R tests/testthat/test-wordcount.R

tokenizers.Rproj

vignettes/introduction-to-tokenizers.R vignettes/introduction-to-tokenizers.Rmd

vignettes/introduction-to-tokenizers.html

vignettes/tif-and-tokenizers.R vignettes/tif-and-tokenizers.Rmd

vignettes/tif-and-tokenizers.html

lmullen/tokenizers documentation built on March 28, 2024, 11:12 a.m.