In kevinushey/sourcetools: Tools for Reading, Tokenizing and Parsing R Code

library(sourcetools)
library(microbenchmark)

sourcetools

Tools for reading, tokenizing, and (eventually) parsing R code.

Getting Started

You can install sourcetools from CRAN with:

install.packages("sourcetools")

Or, you can install the development version from GitHub with:

devtools::install_github("kevinushey/sourcetools")

Reading

sourcetools comes with a couple fast functions for reading files into R.

Use read() and read_lines() to quickly read a file into R as character vectors. read_lines() handles both Windows style \r\n line endings, as well as Unix-style \n endings. Performance is on par with the readers provided by the readr package.

text <- replicate(10000, {
  paste(sample(letters, 200, TRUE), collapse = "")
})
file <- tempfile()
cat(text, file = file, sep = "\n")
mb <- microbenchmark::microbenchmark(times = 10,
  base::readLines(file),
  readr::read_lines(file),
  sourcetools::read_lines(file)
)
sm <- summary(mb)
print(sm[c("expr", "mean", "median")], digits = 3)
unlink(file)

Tokenization

sourcetools provides the tokenize_string() and tokenize_file() functions for generating a tokenized representation of R code. These produce 'raw' tokenized representations of the code, with each token's value as a string, and a recorded row, column, and type: