tokenize_dictionary_values: Internal function to tokenize dictionary values

View source: R/dictionaries.R

tokenize_dictionary_valuesR Documentation

Internal function to tokenize dictionary values

Description

Internal function to tokenize dictionary values

Usage

tokenize_dictionary_values(dict, separator, ...)

Arguments

dict

list of object

Examples

dict <- dictionary(list(ASIA = list("IN" = "\u5370\u5ea6",
                                    "ID" = "\u5370\u5ea6\u5c3c\u897f\u4e9a")))
quanteda:::tokenize_dictionary_values(dict, " ")

quanteda documentation built on April 7, 2026, 1:06 a.m.