tokenlist: Create Token Object
| tokenlist | R Documentation |
Create Token Object
Description
A tokenlist object is a thin wrapper around a list of character vectors, with a few attributes.
Usage
tokenlist(tokens = list(), lemma = NULL, pos = NULL)
Arguments
tokens |
List of character vectors |
lemma |
List of character vectors, must be same size and shape as |
pos |
List of character vectors, must be same size and shape as |
Value
a tokenlist object.
Examples
abc <- list(letters, LETTERS)
tokenlist(abc)
unclass(tokenlist(abc))
tibble(text = tokenlist(abc))
library(tokenizers)
library(modeldata)
data(tate_text)
tokens <- tokenize_words(as.character(tate_text$medium))
tokenlist(tokens)
