default_tokenize: Default function to tokenize

In thomasp85/lime: Local Interpretable Model-Agnostic Explanations

Default function to tokenize

Description

This tokenizer uses stringi::stri_split_boundaries() to tokenize a character vector. To be used with [explain.character()'.

Usage

default_tokenize(text)

Arguments

text	text to tokenize as a character vector

Value

a character vector.

Examples

data('train_sentences')
default_tokenize(train_sentences$text[1])

thomasp85/lime documentation built on Aug. 19, 2022, 5 p.m.