Tokenizer: Tokenizer

Description Usage Arguments Value

View source: R/text_core.R

Description

Provides a consistent 'Transform' interface to tokenizers operating on 'DataFrame's and folders

Usage

1
2
3
4
5
6
7
8
Tokenizer(
  tok,
  rules = NULL,
  counter = NULL,
  lengths = NULL,
  mode = NULL,
  sep = " "
)

Arguments

tok

tokenizer

rules

rules

counter

counter

lengths

lengths

mode

mode

sep

separator

Value

None


fastai documentation built on Oct. 25, 2021, 5:08 p.m.