tokens_restore: Restore special tokens

View source: R/tokens_restore.R

tokens_restoreR Documentation

Restore special tokens

Description

Compounds a sequence of tokens marked by special markers. The beginning and the end of the sequence should be marked by U+E001 and U+E002 respectively.

Usage

tokens_restore(x)

Arguments

x

tokens object


quanteda documentation built on Sept. 11, 2024, 6:08 p.m.