BaseTokenizer: BaseTokenizer

View source: R/text_core.R

BaseTokenizerR Documentation

BaseTokenizer

Description

Basic tokenizer that just splits on spaces

Usage

BaseTokenizer(split_char = " ")

Arguments

split_char

separator

Value

None


fastai documentation built on March 21, 2022, 9:07 a.m.