tokenize_df: Tokenize_df

View source: R/text_core.R

tokenize_dfR Documentation

Tokenize_df

Description

Tokenize texts in 'df[text_cols]' in parallel using 'n_workers'

Usage

tokenize_df(
  df,
  text_cols,
  n_workers = 6,
  rules = NULL,
  mark_fields = NULL,
  tok = NULL,
  tok_text_col = "text"
)

Arguments

df

data frame

text_cols

text columns

n_workers

number of workers

rules

rules

mark_fields

mark_fields

tok

tokenizer

tok_text_col

tok_text_col

Value

None


fastai documentation built on March 21, 2022, 9:07 a.m.