tokens_restore: Restore special tokens

View source: R/tokens_restore.R

tokens_restoreR Documentation

Restore special tokens

Description

Compounds segments of tokens marked by special markers. The beginning and the end of the segments should be marked by U+E001 and U+E002 respectively.

Usage

tokens_restore(x)

Arguments

x

tokens object

Value

a modified tokens object


quanteda documentation built on May 31, 2023, 8:28 p.m.