Man pages for ropensci/tokenizers
Fast, Consistent Tokenization of Natural Language Text

basic-tokenizersBasic tokenizers
chunk_textChunk text into smaller segments
mobydickThe text of Moby Dick
ngram-tokenizersN-gram tokenizers
ptb-tokenizerPenn Treebank Tokenizer
shingle-tokenizersCharacter shingle tokenizers
stem-tokenizersWord stem tokenizer
tokenizersTokenizers
word-countingCount words, sentences, characters
ropensci/tokenizers documentation built on Dec. 24, 2019, 5:11 p.m.