split_token: Split Tokens

In trinker/textshape: Tools for Reshaping Text

Split Tokens

Description

Split tokens.

Usage

split_token(x, ...)

## Default S3 method:
split_token(x, lower = TRUE, ...)

## S3 method for class 'data.frame'
split_token(x, text.var = TRUE, lower = TRUE, ...)

Arguments

x	A data.frame or character vector with tokens.
lower	logical. If TRUE the words are converted to lower case.
text.var	The name of the text variable. If TRUE split_token tries to detect the text column with tokens.
...	Ignored.

Value

Returns a list of vectors of tokens or an expanded data.table with tokens split apart.

Examples

(x <- c(
    "Mr. Brown comes! He says hello. i give him coffee.",
    "I'll go at 5 p. m. eastern time.  Or somewhere in between!",
    "go there"
))
split_token(x)
split_token(x, lower=FALSE)

data(DATA)
split_token(DATA)
split_token(DATA, lower=FALSE)

## Larger data set
split_token(hamlet)

trinker/textshape documentation built on April 5, 2024, 11:39 a.m.