streamingRead: Process a large file piecewise

In taxonomizr: Functions to Work with NCBI Accessions and Taxonomy

Process a large file piecewise

Description

A convenience function to read in a large file piece by piece, process it (hopefully reducing the size either by summarizing or removing extra rows or columns) and return the output

Usage

streamingRead(
  bigFile,
  n = 1e+06,
  FUN = function(xx) sub(",.*", "", xx),
  ...,
  vocal = FALSE
)

Arguments

bigFile	a string giving the path to a file to be read in or a connection opened with "r" mode
n	number of lines to read per chunk
FUN	a function taking the unparsed lines from a chunk of the bigfile as a single argument and returning the desired output
...	any additional arguments to FUN
vocal	if TRUE cat a "." as each chunk is processed

Value

a list containing the results from applying func to the multiple chunks of the file

Examples

tmpFile<-tempfile()
writeLines(LETTERS,tmpFile)
streamingRead(tmpFile,10,head,1)
writeLines(letters,tmpFile)
streamingRead(tmpFile,2,paste,collapse='',vocal=TRUE)
unlist(streamingRead(tmpFile,2,sample,1))

taxonomizr documentation built on Feb. 16, 2023, 6:25 p.m.