TokenizeWithRules: TokenizeWithRules

Description Usage Arguments Value

View source: R/text_core.R

Description

A wrapper around 'tok' which applies 'rules', then tokenizes, then applies 'post_rules'

Usage

1
TokenizeWithRules(tok, rules = NULL, post_rules = NULL)

Arguments

tok

tokenizer

rules

rules

post_rules

post_rules

Value

None


fastai documentation built on July 28, 2021, 5:06 p.m.