tokenize_chinese_chars: Add whitespace around any CJK character.
In jonathanbratt/RBERT: R Implementation of BERT
tokenize_chinese_chars R Documentation
Add whitespace around any CJK character.
Description
(R implementation of BasicTokenizer._tokenize_chinese_chars from
BERT: tokenization.py.) This may result in doubled-up spaces,
but that's the behavior of the python code...
Usage
tokenize_chinese_chars(text)
Arguments
text
A character scalar.
Value
Text with spaces around CJK characters.
jonathanbratt/RBERT documentation built on Jan. 26, 2023, 4:15 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
| tokenize_chinese_chars | R Documentation |
Add whitespace around any CJK character.
Description
(R implementation of BasicTokenizer._tokenize_chinese_chars from BERT: tokenization.py.) This may result in doubled-up spaces, but that's the behavior of the python code...
Usage
tokenize_chinese_chars(text)
Arguments
text |
A character scalar. |
Value
Text with spaces around CJK characters.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.