| step_tokenize_bpe | R Documentation |
step_tokenize_bpe() creates a specification of a recipe step that will
convert a character predictor into a token variable using
Byte Pair Encoding.
step_tokenize_bpe(
recipe,
...,
role = NA,
trained = FALSE,
columns = NULL,
vocabulary_size = 1000,
options = list(),
res = NULL,
skip = FALSE,
id = rand_id("tokenize_bpe")
)
recipe |
A |
... |
One or more selector functions to choose which variables are
affected by the step. See |
role |
Not used by this step since no new variables are created. |
trained |
A logical to indicate if the quantities for preprocessing have been estimated. |
columns |
A character vector of the selected variable names. This is
|
vocabulary_size |
An integer, indicating the number of tokens in the
final vocabulary. Defaults to |
options |
A list of options passed to |
res |
The fitted |
skip |
A logical. Should the step be skipped when the recipe is baked
by |
id |
A character string that is unique to this step to identify it. |
An updated version of recipe with the new step added
to the sequence of existing steps (if any).
When you tidy() this step, a tibble is returned with
columns terms and id:
character, the selectors or variables selected
character, id of this step
This step has 1 tuning parameters:
vocabulary_size: # Unique Tokens in Vocabulary (type: integer, default: 1000)
The underlying operation does not allow for case weights.
step_untokenize() to untokenize.
Other Steps for Tokenization:
step_tokenize(),
step_tokenize_sentencepiece(),
step_tokenize_wordpiece()
library(recipes)
library(modeldata)
data(tate_text)
tate_rec <- recipe(~., data = tate_text) |>
step_tokenize_bpe(medium)
tate_obj <- tate_rec |>
prep()
bake(tate_obj, new_data = NULL, medium) |>
slice(1:2)
bake(tate_obj, new_data = NULL) |>
slice(2) |>
pull(medium)
tidy(tate_rec, number = 1)
tidy(tate_obj, number = 1)
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.