tokenlist: Create Token Object

In EmilHvitfeldt/textrecipes: Extra 'Recipes' for Text Processing

Create Token Object

Description

A tokenlist object is a thin wrapper around a list of character vectors, with a few attributes.

Usage

tokenlist(tokens = list(), lemma = NULL, pos = NULL)

Arguments

tokens	List of character vectors
lemma	List of character vectors, must be same size and shape as x.
pos	List of character vectors, must be same size and shape as x.

Value

a tokenlist object.

Examples

abc <- list(letters, LETTERS)
tokenlist(abc)

unclass(tokenlist(abc))

tibble(text = tokenlist(abc))

library(tokenizers)
library(modeldata)
data(tate_text)
tokens <- tokenize_words(as.character(tate_text$medium))

tokenlist(tokens)

EmilHvitfeldt/textrecipes documentation built on April 7, 2024, 5:02 a.m.