tokenize_csv: Tokenize_csv

Description Usage Arguments Value

View source: R/text_core.R

Description

Tokenize texts in the 'text_cols' of the csv 'fname' in parallel using 'n_workers'

Usage

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
tokenize_csv(
  fname,
  text_cols,
  outname = NULL,
  n_workers = 4,
  rules = NULL,
  mark_fields = NULL,
  tok = NULL,
  header = "infer",
  chunksize = 50000
)

Arguments

fname

file name

text_cols

text columns

outname

outname

n_workers

numeber of workers

rules

rules

mark_fields

mark fields

tok

tokenizer

header

header

chunksize

chunk size

Value

None


fastai documentation built on Oct. 25, 2021, 5:08 p.m.