tokenize_text_by_language: Language-aware tokenizer used across embedders and keyword...
In VectrixDB: Lightweight Vector Database with Embedded Machine Learning Models
tokenize_text_by_language R Documentation
Language-aware tokenizer used across embedders and keyword search
Description
Language-aware tokenizer used across embedders and keyword search
Usage
tokenize_text_by_language(text, language = "en", remove_stopwords = FALSE)
Arguments
text
Input text
language
"en" or "ml"
remove_stopwords
Remove English stopwords when language is "en"
Value
Character vector of tokens
VectrixDB documentation built on Feb. 20, 2026, 5:09 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
| tokenize_text_by_language | R Documentation |
Language-aware tokenizer used across embedders and keyword search
Description
Language-aware tokenizer used across embedders and keyword search
Usage
tokenize_text_by_language(text, language = "en", remove_stopwords = FALSE)
Arguments
text |
Input text |
language |
"en" or "ml" |
remove_stopwords |
Remove English stopwords when language is "en" |
Value
Character vector of tokens
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.