naiveTokenizer: Naive Tokenizer
In LazerLambda/RolliNLP: Natural Language Processing and String Editing Utilities
View source: R/NaiveTokenizer.R
Description
Simple Tokenizer to split words among punctuation and whitespaces.
If possible, prefer a DL Tokenizer.
WARNING: This tokenizer is build for the english language and can be applied
to other latin-based or cyrillic-based languages. This tokenizer does not work on other alphabets
like chinese, devanagari, thai, japanese, hebrew or arabic.
Usage
1
naiveTokenizer(string)
Arguments
string
character string to be tokenized
LazerLambda/RolliNLP documentation built on Oct. 17, 2020, 8:54 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
View source: R/NaiveTokenizer.R
Description
Simple Tokenizer to split words among punctuation and whitespaces. If possible, prefer a DL Tokenizer. WARNING: This tokenizer is build for the english language and can be applied to other latin-based or cyrillic-based languages. This tokenizer does not work on other alphabets like chinese, devanagari, thai, japanese, hebrew or arabic.
Usage
1 | naiveTokenizer(string)
|
Arguments
string |
character string to be tokenized |
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.