as_tokens: Create a list of tokens
In paithiov909/kagomer: An R Wrapper for 'vibrato'

as_tokens

R Documentation

Create a list of tokens

Create a list of tokens

as_tokens(
  tbl,
  token_field = "token",
  pos_field = get_dict_features()[1],
  nm = NULL
)

`tbl`	A tibble of tokens out of `tokenize()`.
`token_field`	<`data-masked`> Column containing tokens.
`pos_field`	Column containing features that will be kept as the names of tokens. If you don't need them, give a `NULL` for this argument.
`nm`	Names of returned list. If left with `NULL`, "doc_id" field of `tbl` is used instead.