Tokenizer: Tokenizer
In fastai: Interface to 'fastai'

View source: R/text_core.R

Tokenizer

R Documentation

Tokenizer

Description

Provides a consistent 'Transform' interface to tokenizers operating on 'DataFrame's and folders

Usage

Tokenizer(
  tok,
  rules = NULL,
  counter = NULL,
  lengths = NULL,
  mode = NULL,
  sep = " "
)

Arguments

`tok`	tokenizer
`rules`	rules
`counter`	counter
`lengths`	lengths
`mode`	mode
`sep`	separator

Value

None

fastai documentation built on March 21, 2022, 9:07 a.m.

fastai index

README.md Audio Classification Basic Image Classification Basic Tabular Bayesian Optimisation Callbacks Custom Image Classification Data augmentation GPT2 Head pose Low-level ops Medical image Migrating from Catalyst Migrating from Ignite Migrating from Lightning Migrating from Pytorch Multilabel classification Object detection Optimizer Question-Answering RoBERTa Speech Recognition Super-Resolution GAN Text-summarization Time-Series

rdrr.io home R language documentation Run R code online

CRAN packages Bioconductor packages R-Forge packages GitHub packages

Note that we can't provide technical support on individual packages. You should contact the package authors for that.

Tweet to @rdrrHQ

GitHub issue tracker

ian@mutexlabs.com