step_tokenize_jp: Tokenization of Japanese character variables
In uribo/washoku: Extra 'recipes' for Japanese Text, Date and Address Processing

View source: R/tokenize.R

step_tokenize_jp

R Documentation

Tokenization of Japanese character variables

Description

step_tokenize_jp() creates a specification of a recipe step that will convert a character predictor into a tokenlist_jp.

Usage

step_tokenize_jp(
  recipe,
  ...,
  role = NA,
  trained = FALSE,
  columns = NULL,
  engine = "sudachir",
  options = list(mode = "A", type = "surface", pos = TRUE, instance = NULL),
  skip = FALSE,
  id = rand_id("tokenize_jp")
)

Arguments

`recipe`	A recipe object. The step will be added to the sequence of operations for this recipe.
`...`	One or more selector functions to choose which variables are affected by the step. See `recipes::selections()` for more details.
`role`	Not used by this step since no new variables are created.
`trained`	A logical to indicate if the quantities for preprocessing have been estimated.
`columns`	A character string of variable names that will be populated (eventually) by the `terms` argument. This is `NULL` until the step is trained by `recipes::prep.recipe()`.
`engine`	Implement token engine package. Defaults to 'sudachir'.
`options`	list. path to engine's function.
`skip`	A logical. Should the step be skipped when the recipe is baked by `recipes::bake.recipe()`? While all operations are baked when `recipes::prep.recipe()` is run, some operations may not be able to be conducted on new data (e.g. processing the outcome variable(s)). Care should be taken when using `skip = FALSE`.
`id`	A character string that is unique to this step to identify it.