step_stopwords | R Documentation |
step_stopwords()
creates a specification of a recipe step that will
filter a token
variable for stop words.
step_stopwords(
recipe,
...,
role = NA,
trained = FALSE,
columns = NULL,
language = "en",
keep = FALSE,
stopword_source = "snowball",
custom_stopword_source = NULL,
skip = FALSE,
id = rand_id("stopwords")
)
recipe |
A recipe object. The step will be added to the sequence of operations for this recipe. |
... |
One or more selector functions to choose which
variables are affected by the step. See |
role |
Not used by this step since no new variables are created. |
trained |
A logical to indicate if the quantities for preprocessing have been estimated. |
columns |
A character string of variable names that will
be populated (eventually) by the |
language |
A character to indicate the language of stop words by ISO 639-1 coding scheme. |
keep |
A logical. Specifies whether to keep the stop words or discard them. |
stopword_source |
A character to indicate the stop words source as
listed in |
custom_stopword_source |
A character vector to indicate a custom list of words that cater to the users specific problem. |
skip |
A logical. Should the step be skipped when the
recipe is baked by |
id |
A character string that is unique to this step to identify it. |
Stop words are words which sometimes are removed before natural language processing tasks. While stop words usually refers to the most common words in the language there is no universal stop word list.
The argument custom_stopword_source
allows you to pass a character vector
to filter against. With the keep
argument one can specify words to keep
instead of removing thus allowing you to select words with a combination of
these two arguments.
An updated version of recipe
with the new step added
to the sequence of existing steps (if any).
When you tidy()
this step, a tibble is returned with
columns terms
, value
, keep
, and id
:
character, the selectors or variables selected
character, name of stop word list
logical, whether stop words are removed or kept
character, id of this step
The underlying operation does not allow for case weights.
step_tokenize()
to turn characters into tokens
Other Steps for Token Modification:
step_lemma()
,
step_ngram()
,
step_pos_filter()
,
step_stem()
,
step_tokenfilter()
,
step_tokenmerge()
library(recipes)
library(modeldata)
data(tate_text)
tate_rec <- recipe(~., data = tate_text) %>%
step_tokenize(medium) %>%
step_stopwords(medium)
tate_obj <- tate_rec %>%
prep()
bake(tate_obj, new_data = NULL, medium) %>%
slice(1:2)
bake(tate_obj, new_data = NULL) %>%
slice(2) %>%
pull(medium)
tidy(tate_rec, number = 2)
tidy(tate_obj, number = 2)
# With a custom stop words list
tate_rec <- recipe(~., data = tate_text) %>%
step_tokenize(medium) %>%
step_stopwords(medium, custom_stopword_source = c("twice", "upon"))
tate_obj <- tate_rec %>%
prep(traimomg = tate_text)
bake(tate_obj, new_data = NULL) %>%
slice(2) %>%
pull(medium)
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.