dot-process_vocab: Process a Vocabulary for Tokenization
In wordpiece: R Implementation of Wordpiece Tokenization
.process_vocab R Documentation
Process a Vocabulary for Tokenization
Description
Process a Vocabulary for Tokenization
Usage
.process_vocab(v)
## Default S3 method:
.process_vocab(v)
## S3 method for class 'wordpiece_vocabulary'
.process_vocab(v)
## S3 method for class 'character'
.process_vocab(v)
Arguments
v
An object of class wordpiece_vocabulary or a character vector.
Value
A character vector of tokens for tokenization.
wordpiece documentation built on March 18, 2022, 5:55 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
| .process_vocab | R Documentation |
Process a Vocabulary for Tokenization
Description
Process a Vocabulary for Tokenization
Usage
.process_vocab(v) ## Default S3 method: .process_vocab(v) ## S3 method for class 'wordpiece_vocabulary' .process_vocab(v) ## S3 method for class 'character' .process_vocab(v)
Arguments
v |
An object of class |
Value
A character vector of tokens for tokenization.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.