tokenlist: Create Token Object

View source: R/tokenlist.R

tokenlistR Documentation

Create Token Object

Description

A tokenlist object is a thin wrapper around a list of character vectors, with a few attributes.

Usage

tokenlist(tokens = list(), lemma = NULL, pos = NULL)

Arguments

tokens

A list of character vectors.

lemma

A list of character vectors, must be the same size and shape as tokens.

pos

A list of character vectors, must be the same size and shape as tokens.

Value

a tokenlist object.

Examples


abc <- list(letters, LETTERS)
tokenlist(abc)

unclass(tokenlist(abc))

tibble(text = tokenlist(abc))

library(tokenizers)
library(modeldata)
data(tate_text)
tokens <- tokenize_words(as.character(tate_text$medium))

tokenlist(tokens)


textrecipes documentation built on Nov. 3, 2026, 5:07 p.m.