tokenlist: Create Token Object
In textrecipes: Extra 'Recipes' for Text Processing

tokenlist

R Documentation

Create Token Object

Description

A tokenlist object is a thin wrapper around a list of character vectors, with a few attributes.

Usage

tokenlist(tokens = list(), lemma = NULL, pos = NULL)

Arguments

`tokens`	List of character vectors
`lemma`	List of character vectors, must be same size and shape as `x`.
`pos`	List of character vectors, must be same size and shape as `x`.

Value

a tokenlist object.

Examples

abc <- list(letters, LETTERS)
tokenlist(abc)

unclass(tokenlist(abc))

tibble(text = tokenlist(abc))

library(tokenizers)
library(modeldata)
data(tate_text)
tokens <- tokenize_words(as.character(tate_text$medium))

tokenlist(tokens)

textrecipes documentation built on Nov. 16, 2023, 5:06 p.m.