tokenizer: Tokenizers
In tm: Text Mining Package
Description
Usage
Arguments
Details
Value
See Also
Examples
Description
Tokenize a document or character vector.
Usage
1
2
3
Boost_tokenizer(x)
MC_tokenizer(x)
scan_tokenizer(x)
Arguments
x
A character vector, or an object that can be coerced to character by
as.character.
Details
The quality and correctness of a tokenization algorithm highly depends
on the context and application scenario. Relevant factors are the
language of the underlying text and the notions of whitespace (which
can vary with the used encoding and the language) and punctuation
marks. Consequently, for superior results you probably need a custom
tokenization function.
- Boost_tokenizer
Uses the Boost (https://www.boost.org)
Tokenizer (via Rcpp).
- MC_tokenizer
Implements the functionality of the tokenizer in the
MC toolkit (https://www.cs.utexas.edu/users/dml/software/mc/).
- scan_tokenizer
Simulates scan(..., what = "character").
Value
A character vector consisting of tokens obtained by tokenization of x.
See Also
getTokenizers to list tokenizers provided by package tm.
Regexp_Tokenizer for tokenizers using regular expressions
provided by package NLP.
tokenize for a simple regular expression based tokenizer
provided by package tau.
tokenizers for a collection of tokenizers provided
by package tokenizers.
Examples
1
2
3
4
5
6
7
data("crude")
Boost_tokenizer(crude[[1]])
MC_tokenizer(crude[[1]])
scan_tokenizer(crude[[1]])
strsplit_space_tokenizer <- function(x)
unlist(strsplit(as.character(x), "[[:space:]]+"))
strsplit_space_tokenizer(crude[[1]])
Example output
Loading required package: NLP
[1] "Diamond" "Shamrock" "Corp" "said" "that"
[6] "effective" "today" "it" "had" "cut"
[11] "its" "contract" "prices" "for" "crude"
[16] "oil" "by" "1.50" "dlrs" "a"
[21] "barrel." "The" "reduction" "brings" "its"
[26] "posted" "price" "for" "West" "Texas"
[31] "Intermediate" "to" "16.00" "dlrs" "a"
[36] "barrel," "the" "copany" "said." "\"The"
[41] "price" "reduction" "today" "was" "made"
[46] "in" "the" "light" "of" "falling"
[51] "oil" "product" "prices" "and" "a"
[56] "weak" "crude" "oil" "market,\"" "a"
[61] "company" "spokeswoman" "said." "Diamond" "is"
[66] "the" "latest" "in" "a" "line"
[71] "of" "U.S." "oil" "companies" "that"
[76] "have" "cut" "its" "contract," "or"
[81] "posted," "prices" "over" "the" "last"
[86] "two" "days" "citing" "weak" "oil"
[91] "markets." "Reuter"
[1] "Diamond" "Shamrock" "Corp" "said" "that"
[6] "effective" "today" "it" "had" "cut"
[11] "its" "contract" "prices" "for" "crude"
[16] "oil" "by" "dlrs" "a" "barrel"
[21] "The" "reduction" "brings" "its" "posted"
[26] "price" "for" "West" "Texas" "Intermediate"
[31] "to" "dlrs" "a" "barrel" "the"
[36] "copany" "said" "The" "price" "reduction"
[41] "today" "was" "made" "in" "the"
[46] "light" "of" "falling" "oil" "product"
[51] "prices" "and" "a" "weak" "crude"
[56] "oil" "market" "a" "company" "spokeswoman"
[61] "said" "Diamond" "is" "the" "latest"
[66] "in" "a" "line" "of" "U"
[71] "S" "oil" "companies" "that" "have"
[76] "cut" "its" "contract" "or" "posted"
[81] "prices" "over" "the" "last" "two"
[86] "days" "citing" "weak" "oil" "markets"
[91] "Reuter"
[1] "Diamond" "Shamrock" "Corp" "said" "that"
[6] "effective" "today" "it" "had" "cut"
[11] "its" "contract" "prices" "for" "crude"
[16] "oil" "by" "1.50" "dlrs" "a"
[21] "barrel." "The" "reduction" "brings" "its"
[26] "posted" "price" "for" "West" "Texas"
[31] "Intermediate" "to" "16.00" "dlrs" "a"
[36] "barrel," "the" "copany" "said." "\"The"
[41] "price" "reduction" "today" "was" "made"
[46] "in" "the" "light" "of" "falling"
[51] "oil" "product" "prices" "and" "a"
[56] "weak" "crude" "oil" "market,\"" "a"
[61] "company" "spokeswoman" "said." "Diamond" "is"
[66] "the" "latest" "in" "a" "line"
[71] "of" "U.S." "oil" "companies" "that"
[76] "have" "cut" "its" "contract," "or"
[81] "posted," "prices" "over" "the" "last"
[86] "two" "days" "citing" "weak" "oil"
[91] "markets." "Reuter"
[1] "Diamond" "Shamrock" "Corp" "said" "that"
[6] "effective" "today" "it" "had" "cut"
[11] "its" "contract" "prices" "for" "crude"
[16] "oil" "by" "1.50" "dlrs" "a"
[21] "barrel." "The" "reduction" "brings" "its"
[26] "posted" "price" "for" "West" "Texas"
[31] "Intermediate" "to" "16.00" "dlrs" "a"
[36] "barrel," "the" "copany" "said." "\"The"
[41] "price" "reduction" "today" "was" "made"
[46] "in" "the" "light" "of" "falling"
[51] "oil" "product" "prices" "and" "a"
[56] "weak" "crude" "oil" "market,\"" "a"
[61] "company" "spokeswoman" "said." "Diamond" "is"
[66] "the" "latest" "in" "a" "line"
[71] "of" "U.S." "oil" "companies" "that"
[76] "have" "cut" "its" "contract," "or"
[81] "posted," "prices" "over" "the" "last"
[86] "two" "days" "citing" "weak" "oil"
[91] "markets." "Reuter"
tm documentation built on April 7, 2021, 3:01 a.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Description Usage Arguments Details Value See Also Examples
Description
Tokenize a document or character vector.
Usage
1 2 3 | Boost_tokenizer(x)
MC_tokenizer(x)
scan_tokenizer(x)
|
Arguments
x |
A character vector, or an object that can be coerced to character by
|
Details
The quality and correctness of a tokenization algorithm highly depends on the context and application scenario. Relevant factors are the language of the underlying text and the notions of whitespace (which can vary with the used encoding and the language) and punctuation marks. Consequently, for superior results you probably need a custom tokenization function.
- Boost_tokenizer
Uses the Boost (https://www.boost.org) Tokenizer (via Rcpp).
- MC_tokenizer
Implements the functionality of the tokenizer in the MC toolkit (https://www.cs.utexas.edu/users/dml/software/mc/).
- scan_tokenizer
Simulates
scan(..., what = "character").
Value
A character vector consisting of tokens obtained by tokenization of x.
See Also
getTokenizers to list tokenizers provided by package tm.
Regexp_Tokenizer for tokenizers using regular expressions
provided by package NLP.
tokenize for a simple regular expression based tokenizer
provided by package tau.
tokenizers for a collection of tokenizers provided
by package tokenizers.
Examples
1 2 3 4 5 6 7 | data("crude")
Boost_tokenizer(crude[[1]])
MC_tokenizer(crude[[1]])
scan_tokenizer(crude[[1]])
strsplit_space_tokenizer <- function(x)
unlist(strsplit(as.character(x), "[[:space:]]+"))
strsplit_space_tokenizer(crude[[1]])
|
Example output
Loading required package: NLP
[1] "Diamond" "Shamrock" "Corp" "said" "that"
[6] "effective" "today" "it" "had" "cut"
[11] "its" "contract" "prices" "for" "crude"
[16] "oil" "by" "1.50" "dlrs" "a"
[21] "barrel." "The" "reduction" "brings" "its"
[26] "posted" "price" "for" "West" "Texas"
[31] "Intermediate" "to" "16.00" "dlrs" "a"
[36] "barrel," "the" "copany" "said." "\"The"
[41] "price" "reduction" "today" "was" "made"
[46] "in" "the" "light" "of" "falling"
[51] "oil" "product" "prices" "and" "a"
[56] "weak" "crude" "oil" "market,\"" "a"
[61] "company" "spokeswoman" "said." "Diamond" "is"
[66] "the" "latest" "in" "a" "line"
[71] "of" "U.S." "oil" "companies" "that"
[76] "have" "cut" "its" "contract," "or"
[81] "posted," "prices" "over" "the" "last"
[86] "two" "days" "citing" "weak" "oil"
[91] "markets." "Reuter"
[1] "Diamond" "Shamrock" "Corp" "said" "that"
[6] "effective" "today" "it" "had" "cut"
[11] "its" "contract" "prices" "for" "crude"
[16] "oil" "by" "dlrs" "a" "barrel"
[21] "The" "reduction" "brings" "its" "posted"
[26] "price" "for" "West" "Texas" "Intermediate"
[31] "to" "dlrs" "a" "barrel" "the"
[36] "copany" "said" "The" "price" "reduction"
[41] "today" "was" "made" "in" "the"
[46] "light" "of" "falling" "oil" "product"
[51] "prices" "and" "a" "weak" "crude"
[56] "oil" "market" "a" "company" "spokeswoman"
[61] "said" "Diamond" "is" "the" "latest"
[66] "in" "a" "line" "of" "U"
[71] "S" "oil" "companies" "that" "have"
[76] "cut" "its" "contract" "or" "posted"
[81] "prices" "over" "the" "last" "two"
[86] "days" "citing" "weak" "oil" "markets"
[91] "Reuter"
[1] "Diamond" "Shamrock" "Corp" "said" "that"
[6] "effective" "today" "it" "had" "cut"
[11] "its" "contract" "prices" "for" "crude"
[16] "oil" "by" "1.50" "dlrs" "a"
[21] "barrel." "The" "reduction" "brings" "its"
[26] "posted" "price" "for" "West" "Texas"
[31] "Intermediate" "to" "16.00" "dlrs" "a"
[36] "barrel," "the" "copany" "said." "\"The"
[41] "price" "reduction" "today" "was" "made"
[46] "in" "the" "light" "of" "falling"
[51] "oil" "product" "prices" "and" "a"
[56] "weak" "crude" "oil" "market,\"" "a"
[61] "company" "spokeswoman" "said." "Diamond" "is"
[66] "the" "latest" "in" "a" "line"
[71] "of" "U.S." "oil" "companies" "that"
[76] "have" "cut" "its" "contract," "or"
[81] "posted," "prices" "over" "the" "last"
[86] "two" "days" "citing" "weak" "oil"
[91] "markets." "Reuter"
[1] "Diamond" "Shamrock" "Corp" "said" "that"
[6] "effective" "today" "it" "had" "cut"
[11] "its" "contract" "prices" "for" "crude"
[16] "oil" "by" "1.50" "dlrs" "a"
[21] "barrel." "The" "reduction" "brings" "its"
[26] "posted" "price" "for" "West" "Texas"
[31] "Intermediate" "to" "16.00" "dlrs" "a"
[36] "barrel," "the" "copany" "said." "\"The"
[41] "price" "reduction" "today" "was" "made"
[46] "in" "the" "light" "of" "falling"
[51] "oil" "product" "prices" "and" "a"
[56] "weak" "crude" "oil" "market,\"" "a"
[61] "company" "spokeswoman" "said." "Diamond" "is"
[66] "the" "latest" "in" "a" "line"
[71] "of" "U.S." "oil" "companies" "that"
[76] "have" "cut" "its" "contract," "or"
[81] "posted," "prices" "over" "the" "last"
[86] "two" "days" "citing" "weak" "oil"
[91] "markets." "Reuter"
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.