tokens_split: Split tokens by a separator pattern

In quanteda: Quantitative Analysis of Textual Data

Split tokens by a separator pattern

Description

Replaces tokens by multiple replacements consisting of elements split by a separator pattern, with the option of retaining the separator. This function effectively reverses the operation of tokens_compound().

Usage

tokens_split(
  x,
  separator = " ",
  valuetype = c("fixed", "regex"),
  remove_separator = TRUE,
  apply_if = NULL,
  verbose = quanteda_options("verbose")
)

Arguments

x	a tokens object
separator	a single-character pattern match by which tokens are separated
valuetype	the type of pattern matching: "glob" for "glob"-style wildcard expressions; "regex" for regular expressions; or "fixed" for exact matching. See valuetype for details.
remove_separator	if TRUE, remove separator from new tokens
apply_if	logical vector of length ndoc(x); documents are modified only when corresponding values are TRUE, others are left unchanged.
verbose	if TRUE print a summary of the object returned by the function: the number of documents and, depending on the object, the number of characters, tokens and types, or features and occurrences. The number of tokens includes paddings, but the number of features excludes them.

Examples

# undo tokens_compound()
toks1 <- tokens("pork barrel is an idiomatic multi-word expression")
tokens_compound(toks1, phrase("pork barrel"))
tokens_compound(toks1, phrase("pork barrel")) |>
    tokens_split(separator = "_")

# similar to tokens(x, remove_hyphen = TRUE) but post-tokenization
toks2 <- tokens("UK-EU negotiation is not going anywhere as of 2018-12-24.")
tokens_split(toks2, separator = "-", remove_separator = FALSE)

quanteda documentation built on Oct. 21, 2026, 5:07 p.m.