tokenize_folder: Tokenize_folder

In EagerAI/fastai: Interface to 'fastai'

Tokenize_folder

Description

Tokenize text files in 'path' in parallel using 'n_workers'

Usage

tokenize_folder(
  path,
  extensions = NULL,
  folders = NULL,
  output_dir = NULL,
  skip_if_exists = TRUE,
  output_names = NULL,
  n_workers = 6,
  rules = NULL,
  tok = NULL,
  encoding = "utf8"
)

Arguments

path	path
extensions	extensions
folders	folders
output_dir	output_dir
skip_if_exists	skip_if_exists
output_names	output_names
n_workers	number of workers
rules	rules
tok	tokenizer
encoding	encoding

Value

None

EagerAI/fastai documentation built on April 16, 2024, 12:01 p.m.