tokens_recompile: Remove redundant or unused tokens IDs

In quanteda: Quantitative Analysis of Textual Data

Remove redundant or unused tokens IDs

Description

This function reassign tokens IDs after changes in removal or transformation of tokens. It update token IDs to ensure that they are unique and dense. Usually, this function is only applied to tokens_xptr objects at the end of text pre-processing.

Usage

tokens_recompile(x, force = FALSE)

Arguments

x	the tokens_xptr object.
force	if TRUE, update tokens IDs even when it may not be necessary.

Examples

toks <- tokens(c(one = "a b c d A B C D",
                 two = "A B C d"), xptr = TRUE)
toks <- tokens_tolower(toks)
types(toks)
types(tokens_recompile(toks))

quanteda documentation built on Aug. 5, 2026, 1:09 a.m.