BaseTokenizer: BaseTokenizer

View source: R/text_core.R

BaseTokenizerR Documentation

BaseTokenizer

Description

Basic tokenizer that just splits on spaces

Usage

BaseTokenizer(split_char = " ")

Arguments

split_char

separator

Value

None


fastai documentation built on June 22, 2024, 11:15 a.m.