rebuild_tokenizer: Rebuild 'Sudachi' tokenizer

In uribo/sudachir: R Interface to 'Sudachi'

Rebuild 'Sudachi' tokenizer

Description

Rebuild 'Sudachi' tokenizer

Usage

rebuild_tokenizer(
  mode = c("C", "B", "A"),
  dict_type = c("core", "small", "full"),
  config_path = NULL
)

Arguments

mode	Split mode (A, B, C)
dict_type	Dictionary type.
config_path	Absolute path to sudachi.json.

Value

Returns a binding to the instance of <sudachipy.tokenizer.Tokenizer>.

Examples

## Not run: 
tokenizer <- rebuild_tokenizer()
tokenize_to_df("Tokyo, Japan", instance = tokenizer)

## End(Not run)

uribo/sudachir documentation built on Feb. 7, 2023, 11:09 a.m.