Man pages for lmullen/tokenizers
Fast, Consistent Tokenization of Natural Language Text
| basic-tokenizers | Basic tokenizers |
| chunk_text | Chunk text into smaller segments |
| mobydick | The text of Moby Dick |
| ngram-tokenizers | N-gram tokenizers |
| ptb-tokenizer | Penn Treebank Tokenizer |
| shingle-tokenizers | Character shingle tokenizers |
| stem-tokenizers | Word stem tokenizer |
| tokenizers | Tokenizers |
| word-counting | Count words, sentences, characters |
