API for tokenizers
Fast, Consistent Tokenization of Natural Language Text
| Global functions | |
|---|---|
| basic-tokenizers | Man page |
| check_input | Source code |
| check_width | Source code |
| chunk_individual_text | Source code |
| chunk_text | Man page Source code |
| corpus_df_as_corpus_vector | Source code |
| count_characters | Man page Source code |
| count_sentences | Man page Source code |
| count_words | Man page Source code |
| generate_ngrams_batch | Source code |
| get_valid_skips | Source code |
| is_corpus_df | Source code |
| mobydick | Man page |
| ngram-tokenizers | Man page |
| remove_stopwords | Source code |
| simplify_list | Source code |
| skip_ngrams_vectorised | Source code |
| tokenize_character_shingles | Man page Source code |
| tokenize_character_shingles.data.frame | Source code |
| tokenize_character_shingles.default | Source code |
| tokenize_characters | Man page Source code |
| tokenize_characters.data.frame | Source code |
| tokenize_characters.default | Source code |
| tokenize_lines | Man page Source code |
| tokenize_lines.data.frame | Source code |
| tokenize_lines.default | Source code |
| tokenize_ngrams | Man page Source code |
| tokenize_ngrams.data.frame | Source code |
| tokenize_ngrams.default | Source code |
| tokenize_paragraphs | Man page Source code |
| tokenize_paragraphs.data.frame | Source code |
| tokenize_paragraphs.default | Source code |
| tokenize_ptb | Man page Source code |
| tokenize_ptb.data.frame | Source code |
| tokenize_ptb.default | Source code |
| tokenize_regex | Man page Source code |
| tokenize_regex.data.frame | Source code |
| tokenize_regex.default | Source code |
| tokenize_sentences | Man page Source code |
| tokenize_sentences.data.frame | Source code |
| tokenize_sentences.default | Source code |
| tokenize_skip_ngrams | Man page Source code |
| tokenize_skip_ngrams.data.frame | Source code |
| tokenize_skip_ngrams.default | Source code |
| tokenize_word_stems | Man page Source code |
| tokenize_word_stems.data.frame | Source code |
| tokenize_word_stems.default | Source code |
| tokenize_words | Man page Source code |
| tokenize_words.data.frame | Source code |
| tokenize_words.default | Source code |
| tokenizers | Man page |
