TokenizeWithRules: TokenizeWithRules

View source: R/text_core.R

TokenizeWithRulesR Documentation

TokenizeWithRules

Description

A wrapper around 'tok' which applies 'rules', then tokenizes, then applies 'post_rules'

Usage

TokenizeWithRules(tok, rules = NULL, post_rules = NULL)

Arguments

tok

tokenizer

rules

rules

post_rules

post_rules

Value

None


fastai documentation built on March 31, 2023, 11:41 p.m.