| step_tf | R Documentation |
step_tf() creates a specification of a recipe step that will convert a
token variable into multiple variables containing the token
counts.
step_tf(
recipe,
...,
role = "predictor",
trained = FALSE,
columns = NULL,
weight_scheme = "raw count",
weight = 0.5,
vocabulary = NULL,
res = NULL,
prefix = "tf",
sparse = "auto",
keep_original_cols = FALSE,
skip = FALSE,
id = rand_id("tf")
)
recipe |
A |
... |
One or more selector functions to choose which variables are
affected by the step. See |
role |
For model terms created by this step, what analysis role should they be assigned? By default, the new columns created by this step from the original variables will be used as predictors in a model. |
trained |
A logical to indicate if the quantities for preprocessing have been estimated. |
columns |
A character vector of the selected variable names. This is
|
weight_scheme |
A single string determining the weighting scheme for the
term frequency calculations. Must be one of |
weight |
A single numeric weight used if |
vocabulary |
A character vector of the tokens to create variables for.
If |
res |
A named list of the tokens used for each selected variable. This
is |
prefix |
A character string that will be the prefix to the resulting new variables. See notes below. |
sparse |
A single string. Should the columns produced be sparse
vectors? Can take the values |
keep_original_cols |
A logical to keep the original variables in the
output. Defaults to |
skip |
A logical. Should the step be skipped when the recipe is baked
by |
id |
A character string that is unique to this step to identify it. |
It is strongly advised to use step_tokenfilter before using step_tf to limit the number of variables created, otherwise you might run into memory issues. A good strategy is to start with a low token count and go up according to how much RAM you want to use.
Term frequency is a weight of how many times each token appears in each
observation. There are different ways to calculate the weight and this step
can do it in a couple of ways. Setting the argument weight_scheme to
"binary" will result in a set of binary variables denoting if a token is
present in the observation. "raw count" will count the times a token is
present in the observation. "term frequency" will divide the count by the
total number of in-vocabulary tokens in the document to limit the effect
of the document length as longer documents tends to have the word present
more times but not necessarily at a higher percentage. "log normalization"
takes the log of 1 plus the count, adding 1 is done to avoid taking log of
0. Finally "double normalization" is the raw frequency divided by the raw
frequency of the most occurring term in the document. This is then multiplied by weight and
weight is added to the result. This is again done to prevent a bias towards
longer documents.
The new columns will have names that begin with prefix, then the name of
the variable, then the token, all separated by _. For example, with the
default prefix = "tf", a variable named text, and the token "word", the
new column will be named tf_text_word.
An updated version of recipe with the new step added
to the sequence of existing steps (if any).
When you tidy() this step, a tibble is returned with
columns terms, value, and id:
character, the selectors or variables selected
character, the weighting scheme
character, id of this step
This step has 2 tuning parameters:
weight_scheme: Term Frequency Weight Method (type: character, default: raw count)
weight: Weight (type: double, default: 0.5)
This step produces sparse columns if sparse = "yes" is being set. The
default value "auto" won't trigger production of sparse columns if a recipe
is recipes::prep()ed, but allows for a workflow to toggle to "yes" or
"no" depending on whether the model supports recipes::sparse_data and if
the model is expected to run faster with the data.
The mechanism for determining how much sparsity is produced isn't perfect,
and there will be times when you want to manually overwrite by setting
sparse = "yes" or sparse = "no".
The underlying operation does not allow for case weights.
step_tokenize() to turn characters into tokens
Other Steps for Numeric Variables From Tokens:
step_lda(),
step_texthash(),
step_tfidf(),
step_word_embeddings()
library(recipes)
library(modeldata)
data(tate_text)
tate_rec <- recipe(~., data = tate_text) |>
step_tokenize(medium) |>
step_tf(medium)
tate_obj <- tate_rec |>
prep()
bake(tate_obj, tate_text)
tidy(tate_rec, number = 2)
tidy(tate_obj, number = 2)
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.