tokenize_csv: Tokenize_csv
In fastai: Interface to 'fastai'

View source: R/text_core.R

tokenize_csv

R Documentation

Tokenize_csv

Description

Tokenize texts in the 'text_cols' of the csv 'fname' in parallel using 'n_workers'

Usage

tokenize_csv(
  fname,
  text_cols,
  outname = NULL,
  n_workers = 4,
  rules = NULL,
  mark_fields = NULL,
  tok = NULL,
  header = "infer",
  chunksize = 50000
)

Arguments

`fname`	file name
`text_cols`	text columns
`outname`	outname
`n_workers`	numeber of workers
`rules`	rules
`mark_fields`	mark fields
`tok`	tokenizer
`header`	header
`chunksize`	chunk size

Value

None

fastai documentation built on March 31, 2023, 11:41 p.m.

fastai index

README.md Audio Classification Basic Image Classification Basic Tabular Bayesian Optimisation Callbacks Custom Image Classification Data augmentation GPT2 Head pose Low-level ops Medical image Migrating from Catalyst Migrating from Ignite Migrating from Lightning Migrating from Pytorch Multilabel classification Object detection Optimizer Question-Answering RoBERTa Speech Recognition Super-Resolution GAN Text-summarization Time-Series

rdrr.io home R language documentation Run R code online

CRAN packages Bioconductor packages R-Forge packages GitHub packages

Note that we can't provide technical support on individual packages. You should contact the package authors for that.

Tweet to @rdrrHQ

GitHub issue tracker

ian@mutexlabs.com