char_tokenizer: Returns vector of tokens, tokenized by character

Description Usage Arguments Value Examples

View source: R/tokenization.R

Description

Returns vector of tokens, tokenized by character

Usage

1

Arguments

input

String to tokenize

Value

vector of characters

Examples

1
2
char_tokenizer('hello')
# c("h", "e", "l", "l", "o")

d0rj/pido documentation built on Dec. 19, 2021, 8 p.m.