default_tokenize: Default function to tokenize

Description Usage Arguments Value Examples

View source: R/character.R

Description

This tokenizer uses stringi::stri_split_boundaries() to tokenize a character vector. To be used with [explain.character()'.

Usage

1

Arguments

text

text to tokenize as a character vector

Value

a character vector.

Examples

1
2
data('train_sentences')
default_tokenize(train_sentences$text[1])

lime documentation built on Feb. 24, 2021, 5:08 p.m.