BaseTokenizer: BaseTokenizer

Description Usage Arguments Value

View source: R/text_core.R

Description

Basic tokenizer that just splits on spaces

Usage

1
BaseTokenizer(split_char = " ")

Arguments

split_char

separator

Value

None


fastai documentation built on July 28, 2021, 5:06 p.m.