View source: R/token_replace.R
| step_token_replace | R Documentation |
step_token_replace() creates a specification of a recipe step that will
replace tokens in a token variable according to a table of
patterns and replacements.
step_token_replace(
recipe,
...,
role = NA,
trained = FALSE,
columns = NULL,
pattern,
replacement,
fixed = TRUE,
skip = FALSE,
id = rand_id("token_replace")
)
recipe |
A |
... |
One or more selector functions to choose which variables are
affected by the step. See |
role |
Not used by this step since no new variables are created. |
trained |
A logical to indicate if the quantities for preprocessing have been estimated. |
columns |
A character vector of the selected variable names. This is
|
pattern |
A character vector of tokens to look for. Must be the same
length as |
replacement |
A character vector of the tokens to replace the matches with. |
fixed |
Logical. If |
skip |
A logical. Should the step be skipped when the recipe is baked
by |
id |
A character string that is unique to this step to identify it. |
The whole token is replaced, not just the part that matched. If a token
matches more than one element of pattern, the first match is used. Since
the number of tokens is unchanged, the lemma and part-of-speech attributes
of the tokens are kept as is.
This is useful for normalizing spelling variants, mapping tokens to a
placeholder such as "<number>", or applying a small dictionary of
corrections.
An updated version of recipe with the new step added
to the sequence of existing steps (if any).
When you tidy() this step, a tibble is returned
with columns terms, pattern, replacement, and id:
character, the selectors or variables selected
character, the patterns to look for
character, the replacements
character, id of this step
The underlying operation does not allow for case weights.
step_tokenize() to turn characters into tokens
Other Steps for Token Modification:
step_lemma(),
step_ngram(),
step_pos_filter(),
step_stem(),
step_stopwords(),
step_tokenfilter(),
step_tokenmerge()
library(recipes)
library(modeldata)
data(tate_text)
tate_rec <- recipe(~., data = tate_text) |>
step_tokenize(medium) |>
step_token_replace(
medium,
pattern = c("paper", "canvas"),
replacement = c("PAPER", "CANVAS")
)
tate_obj <- tate_rec |>
prep()
bake(tate_obj, new_data = NULL) |>
slice(1:2) |>
pull(medium)
tidy(tate_rec, number = 2)
tidy(tate_obj, number = 2)
# Using regular expressions
tate_rec <- recipe(~., data = tate_text) |>
step_tokenize(medium) |>
step_token_replace(
medium,
pattern = "^[0-9]+$",
replacement = "<number>",
fixed = FALSE
)
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.