nlp_recursive_tokenizer: Spark NLP RecursiveTokenizer

View source: R/recursive-tokenizer.R

nlp_recursive_tokenizerR Documentation

Spark NLP RecursiveTokenizer

Description

Spark ML model that tokenizes See https://nlp.johnsnowlabs.com/api/index#com.johnsnowlabs.nlp.annotators.RecursiveTokenizer

Usage

nlp_recursive_tokenizer(
  x,
  input_cols,
  output_col,
  infixes = NULL,
  prefixes = NULL,
  suffixes = NULL,
  white_list = NULL,
  uid = random_string("recursive_tokenizer_")
)

Arguments

x

A spark_connection, ml_pipeline, or a tbl_spark.

input_cols

Input columns. String array.

output_col

Output column. String.

infixes

strings that will be split when found at the middle of a token

prefixes

strings that will be split when found at the beginning of a token

suffixes

strings that will be split when found at the end of a token

white_list

whitelist

uid

A character string used to uniquely identify the ML estimator.

Value

The object returned depends on the class of x.

  • spark_connection: When x is a spark_connection, the function returns an instance of a ml_estimator object. The object contains a pointer to a Spark Estimator object and can be used to compose Pipeline objects.

  • ml_pipeline: When x is a ml_pipeline, the function returns a ml_pipeline with the NLP estimator appended to the pipeline.

  • tbl_spark: When x is a tbl_spark, an estimator is constructed then immediately fit with the input tbl_spark, returning an NLP model.


r-spark/sparknlp documentation built on Oct. 15, 2022, 10:50 a.m.