tokenize: Obtain token table from text
In krlmlr/styler: Non-Invasive Pretty Printing of R Code
tokenize R Documentation
Obtain token table from text
Description
utils::getParseData() is used to obtain a flat parse table from text.
Usage
tokenize(text)
Arguments
text
The text to parse.
Details
Apart from the columns provided by utils::getParseData(), the following
columns are added:
A column "short" with the first five characters of "text".
A column "pos_id" for (positional id) which can be used for sorting
(because "id" cannot be used in general). Note that the nth value of this
column corresponds to n as long as no tokens are inserted.
A column "child" that contains nests.
Value
A flat parse table
krlmlr/styler documentation built on Dec. 12, 2024, 8:36 a.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
| tokenize | R Documentation |
Obtain token table from text
Description
utils::getParseData() is used to obtain a flat parse table from text.
Usage
tokenize(text)
Arguments
text |
The text to parse. |
Details
Apart from the columns provided by utils::getParseData(), the following
columns are added:
A column "short" with the first five characters of "text".
A column "pos_id" for (positional id) which can be used for sorting (because "id" cannot be used in general). Note that the nth value of this column corresponds to n as long as no tokens are inserted.
A column "child" that contains nests.
Value
A flat parse table
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.