text_tokenize: generic for gregexpr wrappers to tokenize text

In stringb: Convenient Base R String Handling

Description

generic for gregexpr wrappers to tokenize text

default method for text_tokenize generic

Usage

text_tokenize(
  string,
  regex = NULL,
  ignore.case = FALSE,
  fixed = FALSE,
  perl = FALSE,
  useBytes = FALSE,
  non_token = FALSE
)

## Default S3 method:
text_tokenize(
  string,
  regex = NULL,
  ignore.case = FALSE,
  fixed = FALSE,
  perl = FALSE,
  useBytes = FALSE,
  non_token = FALSE
)

Arguments

string	text to be tokenized
regex	regex expressing where to cut see (see grep)
ignore.case	whether or not reges should be case sensitive (see grep)
fixed	whether or not regex should be interpreted as is or as regular expression (see grep)
perl	whether or not Perl compatible regex should be used (see grep)
useBytes	byte-by-byte matching of regex or character-by-character (see grep)
non_token	should information for non-token, i.e. those patterns by which the text was splitted, be returned as well

stringb documentation built on Jan. 26, 2021, 1:07 a.m.