tokenlist: Create tokenlist object

Description Usage Arguments Value Examples

View source: R/tokenlist.R

Description

A tokenlist object is a thin wrapper around a list of character vectors, with a few attributes.

Usage

1
tokenlist(tokens = list(), lemma = NULL, pos = NULL)

Arguments

tokens

List of character vectors

lemma

List of character vectors, must be same size and shape as x.

pos

List of character vectors, must be same size and shape as x.

Value

a tokenlist object.

Examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
abc <- list(letters, LETTERS)
tokenlist(abc)

unclass(tokenlist(abc))

tibble(text = tokenlist(abc))

library(tokenizers)
library(modeldata)
data(okc_text)
tokens <- tokenize_words(okc_text$essay0)

tokenlist(tokens)

textrecipes documentation built on July 11, 2021, 9:06 a.m.