tokenizers: Unicode Alphabetic Tokenizer
In Unicode: Unicode Data and Utilities
tokenizers R Documentation
Unicode Alphabetic Tokenizer
Description
A simple Unicode alphabetic tokenizer.
Usage
Unicode_alphabetic_tokenizer(x)
Arguments
x
a character vector.
Details
Tokenization first replaces the elements of x by their Unicode
character sequences. Then, the non-alphabetic characters (i.e., the
ones which do not have the Alphabetic property) are replaced by
blanks, and the corresponding strings are split according to the
blanks.
Value
A character vector with the tokenized strings.
Unicode documentation built on May 29, 2024, 2:36 a.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
| tokenizers | R Documentation |
Unicode Alphabetic Tokenizer
Description
A simple Unicode alphabetic tokenizer.
Usage
Unicode_alphabetic_tokenizer(x)
Arguments
x |
a character vector. |
Details
Tokenization first replaces the elements of x by their Unicode
character sequences. Then, the non-alphabetic characters (i.e., the
ones which do not have the Alphabetic property) are replaced by
blanks, and the corresponding strings are split according to the
blanks.
Value
A character vector with the tokenized strings.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.