BaseTokenizer: BaseTokenizer

View source: R/text_core.R

BaseTokenizerR Documentation

BaseTokenizer

Description

Basic tokenizer that just splits on spaces

Usage

BaseTokenizer(split_char = " ")

Arguments

split_char

separator

Value

None


fastai documentation built on March 31, 2023, 11:41 p.m.