collapse_tokens: Collapse sequences of tokens by condition

In paithiov909/audubon: Japanese Text Processing Tools

Collapse sequences of tokens by condition

Description

Concatenates sequences of tokens in the tidy text dataset, while grouping them by an expression.

Usage

collapse_tokens(tbl, condition, .collapse = "")

Arguments

tbl	A tidy text dataset.
condition	<data-masked> A logical expression.
.collapse	String with which tokens are concatenated.

Details

Note that this function drops all columns except but 'token' and columns for grouping sequences. So, the returned data.frame has only 'doc_id', 'sentence_id', 'token_id', and 'token' columns.

Value

A data.frame.

Examples

df <- prettify(head(hiroba), col_select = "POS1")
collapse_tokens(df, POS1 == "\u540d\u8a5e")

paithiov909/audubon documentation built on Sept. 28, 2024, 8:47 a.m.