dot-tokenize: tokenize the character vector and prepend the [CLS] token to...

In cregouby/docformer: Fit 'DocFormer' Models for Classification and Regression

tokenize the character vector and prepend the [CLS] token to first

Description

tokenize the character vector and prepend the [CLS] token to first

Usage

.tokenize(tokenizer, x, ...)

Arguments

tokenizer	the tokenizer function
x	character vector to encode
...	may include max_seq_len in future releases. Currently unused

Value

list of token ids for each token

cregouby/docformer documentation built on May 27, 2023, 11:19 p.m.