tokenize: Tokenizing Japanese text

Description Usage Arguments Details Value Examples

Description

Tokenizing Japanese text

Usage

1
2
## S4 method for signature 'tiny_segmenter'
tokenize(text)

Arguments

text

input single character vector

Details

Given a 'text' string, 'tokenize' attempts to segment using the TinySegmenter algorithm.

Value

segment Japanese text

Examples

1
2
3
4
5
6
7
library(rTinySegmenter)
segmentr <- tiny_segmenter$new(text = "私の名前は中野です")
tokenized_text <- segmentr$tokenize()
# feature
segmentr$model_attr$unit
# feature value
segmentr$model_attr$score

yamano357/rTinySegmenter documentation built on May 4, 2019, 2:28 p.m.