#| include: false knitr::opts_chunk$set( collapse = TRUE, comment = "#>" )
#| label: setup library(textrecipes) library(recipes)
Tokenization is the first step in almost every textrecipes workflow. It splits a character column into smaller pieces, called tokens, that later steps can filter, modify, and turn into numeric features. This vignette gives an overview of the tokenization options in the package and how each choice changes the tokens you get.
We will use two short texts throughout.
text_tibble <- tibble::tibble( text = c( "This is a sentence. It has two!", "They're e-mailing Dr. Smith about the 3 cats." ) )
step_tokenize() converts a character column into a token column (see ?tokenlist). The token argument picks the unit of tokenization and the engine argument picks the package doing the work. With no arguments you get word tokens from the tokenizers package.
recipe(~text, data = text_tibble) |> step_tokenize(text) |> show_tokens(text, n = 2)
show_tokens() is a convenient way to see what a recipe produces. Each element of the result is the token vector for one row.
The engine determines which values of token are available.
| Engine | Tokens | Trained on data |
|:-----------------|:-------------------------------------------------------------------------------------------------------------------------------|:----------------|
| tokenizers | "words", "characters", "character_shingle", "ngrams", "skip_ngrams", "sentences", "lines", "paragraphs", "regex", "ptb", "word_stems" | No |
| spacyr | "words" | No |
| tokenizers.bpe | "words" | Yes |
| udpipe | "words" | No (needs a pre-trained model) |
Subword tokenizers such as wordpiece, sentencepiece, and byte pair encoding also have dedicated steps: step_tokenize_wordpiece(), step_tokenize_sentencepiece(), and step_tokenize_bpe().
The tokenizers engine is the default. Every token option corresponds to a function in the tokenizers package, and the options argument takes a named list of arguments passed to that function.
Word tokenization lowercases the text and removes punctuation by default.
recipe(~text, data = text_tibble) |> step_tokenize(text, token = "words") |> show_tokens(text, n = 2)
Both behaviors are controlled through options, see ?tokenizers::tokenize_words. Here we keep the original case and the punctuation.
recipe(~text, data = text_tibble) |> step_tokenize( text, options = list(lowercase = FALSE, strip_punct = FALSE) ) |> show_tokens(text, n = 2)
Note that "e-mailing" is split at the hyphen into "e" and "mailing" by default. Numbers can be removed by setting strip_numeric = TRUE.
recipe(~text, data = text_tibble) |> step_tokenize(text, options = list(strip_numeric = TRUE)) |> show_tokens(text, n = 2)
The "ptb" tokenizer follows the Penn Treebank conventions. It splits contractions such as "They're" into "They" and "'re" and keeps hyphenated words intact, which is often what you want for linguistic work.
recipe(~text, data = text_tibble) |> step_tokenize(text, token = "ptb") |> show_tokens(text, n = 2)
"characters" returns single characters, and "character_shingle" returns overlapping character n-grams whose size is set with n.
recipe(~text, data = text_tibble) |> step_tokenize(text, token = "characters") |> show_tokens(text, n = 2)
recipe(~text, data = text_tibble) |> step_tokenize( text, token = "character_shingle", options = list(n = 3) ) |> show_tokens(text, n = 2)
Character-level tokens are useful for short strings with typos or without clear word boundaries, but they are rarely appropriate for long documents.
"ngrams" returns word n-grams. Use n and n_min to choose the range of sizes.
recipe(~text, data = text_tibble) |> step_tokenize(text, token = "ngrams", options = list(n = 2, n_min = 1)) |> show_tokens(text, n = 2)
"skip_ngrams" allows gaps between the words, controlled by k.
recipe(~text, data = text_tibble) |> step_tokenize( text, token = "skip_ngrams", options = list(n = 2, n_min = 2, k = 1) ) |> show_tokens(text, n = 2)
If you want n-grams of tokens you have already modified, for example after removing stop words, tokenize to words first and use step_ngram() instead. See vignette("Working-with-n-grams").
These tokenizers split at larger boundaries. They are mostly useful as a first step before splitting further, or when each sentence or paragraph is the unit you want to model.
recipe(~text, data = text_tibble) |> step_tokenize(text, token = "sentences") |> show_tokens(text, n = 2)
"lines" splits on line breaks and "paragraphs" on blank lines.
"regex" splits the text wherever a pattern matches. The pattern is given with pattern, and the default is a single space.
recipe(~text, data = text_tibble) |> step_tokenize(text, token = "regex", options = list(pattern = "[.!]\\s*")) |> show_tokens(text, n = 2)
"word_stems" tokenizes into words and stems each of them in one go. If you want to control the stemmer, or stem after other filtering, tokenize to words and use step_stem().
recipe(~text, data = text_tibble) |> step_tokenize(text, token = "word_stems") |> show_tokens(text, n = 2)
Some tokenizers learn their vocabulary from the data. Their training arguments are passed in training_options, while options is used when the tokenizer is applied. The tokenizers.bpe engine learns a byte pair encoding on the training set, and vocab_size is the most important training option. It is usually in the thousands, but is kept small here to fit our tiny data.
#| eval: false recipe(~text, data = text_tibble) |> step_tokenize( text, engine = "tokenizers.bpe", training_options = list(vocab_size = 40) ) |> show_tokens(text, n = 2)
Because the vocabulary is learned during prep(), the same tokenizer is applied to new data when you bake(). The dedicated steps step_tokenize_bpe(), step_tokenize_sentencepiece(), and step_tokenize_wordpiece() give you the same kind of subword tokens with arguments specific to each method.
The spacyr engine tokenizes with spaCy and requires a working Python installation with spaCy. Its options are applied each time the data is tokenized, including at bake() time.
#| eval: false recipe(~text, data = text_tibble) |> step_tokenize(text, engine = "spacyr") |> show_tokens(text, n = 2)
The udpipe engine requires a pre-trained model, loaded with udpipe::udpipe_load_model(), passed as model in training_options.
#| eval: false model <- udpipe::udpipe_load_model("english-ewt-ud-2.5-191206.udpipe") recipe(~text, data = text_tibble) |> step_tokenize( text, engine = "udpipe", training_options = list(model = model) ) |> show_tokens(text, n = 2)
When none of the engines do what you need, pass your own function to custom_token. It must take a character vector and return a list of character vectors with one element per input.
space_tokenizer <- function(x) { strsplit(x, " +") } recipe(~text, data = text_tibble) |> step_tokenize(text, custom_token = space_tokenizer) |> show_tokens(text, n = 2)
There is rarely a single correct choice, and the best tokenizer depends on your data and model. Some rules of thumb:
"words". It is fast, easy to interpret, and works well for many bag-of-words models."ptb" or the spacyr and udpipe engines when linguistic detail such as contractions or part of speech matters.Since the tokenizer is a recipe argument, token can also be tuned with the tunable method for step_tokenize().
After tokenizing you will typically filter and modify the tokens, for example with step_stopwords(), step_stem(), step_tokenfilter(), or step_ngram(), and then turn them into numeric features with step_tf(), step_tfidf(), or step_texthash(). See vignette("cookbook---using-more-complex-recipes-involving-text") for complete examples.
Any scripts or data that you put into this service are public.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.