| step_stopwords | R Documentation |
step_stopwords() creates a specification of a recipe step that will
filter a token variable for stop words.
step_stopwords(
recipe,
...,
role = NA,
trained = FALSE,
columns = NULL,
language = "en",
keep = FALSE,
stopword_source = "snowball",
custom_stopword_source = NULL,
skip = FALSE,
id = rand_id("stopwords")
)
recipe |
A |
... |
One or more selector functions to choose which variables are
affected by the step. See |
role |
Not used by this step since no new variables are created. |
trained |
A logical to indicate if the quantities for preprocessing have been estimated. |
columns |
A character vector of the selected variable names. This is
|
language |
A single string indicating the language of the stop words by
ISO 639-1 coding scheme. Ignored if |
keep |
A logical. Should the stop words be kept instead of discarded?
Defaults to |
stopword_source |
A single string indicating the stop words source, as
listed in |
custom_stopword_source |
A character vector of words to use as a custom
list of stop words, which cater to the user's specific problem. If
|
skip |
A logical. Should the step be skipped when the recipe is baked
by |
id |
A character string that is unique to this step to identify it. |
Stop words are words which sometimes are removed before natural language processing tasks. While stop words usually refers to the most common words in the language there is no universal stop word list.
The argument custom_stopword_source allows you to pass a character vector
to filter against. With the keep argument one can specify words to keep
instead of removing thus allowing you to select words with a combination of
these two arguments.
An updated version of recipe with the new step added
to the sequence of existing steps (if any).
When you tidy() this step, a tibble is returned with
columns terms, value, keep, and id:
character, the selectors or variables selected
character, name of stop word list
logical, whether stop words are removed or kept
character, id of this step
The underlying operation does not allow for case weights.
step_tokenize() to turn characters into tokens
Other Steps for Token Modification:
step_lemma(),
step_ngram(),
step_pos_filter(),
step_stem(),
step_token_replace(),
step_tokenfilter(),
step_tokenmerge()
library(recipes)
library(modeldata)
data(tate_text)
tate_rec <- recipe(~., data = tate_text) |>
step_tokenize(medium) |>
step_stopwords(medium)
tate_obj <- tate_rec |>
prep()
bake(tate_obj, new_data = NULL, medium) |>
slice(1:2)
bake(tate_obj, new_data = NULL) |>
slice(2) |>
pull(medium)
tidy(tate_rec, number = 2)
tidy(tate_obj, number = 2)
# With a custom stop words list
tate_rec <- recipe(~., data = tate_text) |>
step_tokenize(medium) |>
step_stopwords(medium, custom_stopword_source = c("twice", "upon"))
tate_obj <- tate_rec |>
prep(traimomg = tate_text)
bake(tate_obj, new_data = NULL) |>
slice(2) |>
pull(medium)
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.