as_tokens: Create a list of tokens

In uribo/sudachir: R Interface to 'Sudachi'

Create a list of tokens

Description

Create a list of tokens

Usage

as_tokens(tbl, type, pos = TRUE, ...)

Arguments

tbl	A data.frame of tokens out of tokenize_to_df().
type	Preference for the format of returned tokens. Pick one of "surface", "dictionary", "normalized", or "reading".
pos	When passed as TRUE, this function uses the part-of-speech information as the name of the returned tokens.
...	Passed to dict_features().

Value

A named list of character vectors.

Examples

## Not run: 
tokenize_to_df("Tokyo, Japan") |>
  as_tokens(type = "surface")

## End(Not run)

uribo/sudachir documentation built on Feb. 7, 2023, 11:09 a.m.