dot-whitespace_tokenize: Run basic whitespace cleaning and splitting on a piece of...

Description Usage Arguments Value

Description

Run basic whitespace cleaning and splitting on a piece of text.

Usage

1

Arguments

text

Character scalar to tokenize.

Value

Character vector of tokens, split on whitespace.


wordpiece documentation built on Feb. 11, 2021, 5:06 p.m.