#| include: false knitr::opts_chunk$set( collapse = TRUE, comment = "#>" )
#| label: setup library(textrecipes) library(recipes)
This vignette walks through getting a document-term matrix, such as one to hand to glmnet, xgboost, a clustering routine, or your own code, straight out of a recipe. Build a recipe, prep() it on your training text, then bake() with composition = "dgCMatrix".
We will use a handful of short documents.
train <- tibble::tibble( text = c( "the cat sat on the mat", "the dog chased the cat", "dogs and cats are pets", "a mat is a small rug" ) ) new_docs <- tibble::tibble( text = c("the cat chased a small dog", "a rug for the mat") )
Set sparse = "yes" in the step to get sparse output.
rec <- recipe(~text, data = train) |> step_tokenize(text) |> step_tfidf(text, sparse = "yes") |> prep()
Baking the training data with composition = "dgCMatrix" returns a sparse matrix from the Matrix package, one row per document and one column per token.
x_train <- bake(rec, new_data = NULL, composition = "dgCMatrix") x_train
The prepped recipe remembers the vocabulary and the inverse document frequencies learned from the training data, so new documents are mapped onto the same columns.
x_new <- bake(rec, new_data = new_docs, composition = "dgCMatrix") x_new identical(colnames(x_train), colnames(x_new))
Tokens that weren't seen during prep() are dropped, so the columns always line up between the training and new data.
The result is a regular dgCMatrix, which is the format that many R packages take as input.
#| eval: false # Fit a lasso model with glmnet fit <- glmnet::glmnet(x_train, y_train, family = "binomial") predict(fit, newx = x_new)
These steps have a sparse argument, which needs to be set to "yes" to get sparse output:
step_tf(): term frequenciesstep_tfidf(): term frequency-inverse document frequencystep_texthash(): feature hashing of tokensstep_dummy_hash(): feature hashing of nominal variablesA vocabulary can grow very large, so you will often want to put a cap on the number of columns. There are two options.
Use step_tokenfilter() to keep only the most frequent tokens.
#| eval: false recipe(~text, data = large_corpus) |> step_tokenize(text) |> step_tokenfilter(text, max_tokens = 5000) |> step_tf(text, sparse = "yes") |> prep() |> bake(new_data = NULL, composition = "dgCMatrix")
Or use feature hashing with step_texthash(). The number of columns is fixed up front with num_terms, no matter how large the vocabulary is, and no vocabulary has to be stored. This makes it a good match for sparse output when the vocabulary is too big to hold in memory.
#| eval: false recipe(~text, data = large_corpus) |> step_tokenize(text) |> step_texthash(text, num_terms = 4096, sparse = "yes") |> prep() |> bake(new_data = NULL, composition = "dgCMatrix")
step_dummy_hash() does the same for nominal predictors with many levels.
If you are fitting a model with tidymodels, you don't need to set sparse yourself. A workflow turns it on for models that support sparse data. See ?sparse_data in the recipes package for details.
Any scripts or data that you put into this service are public.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.