process_blocks: Blockwise processing of file

In djvanderlaan/LaF: Fast Access to Large ASCII Files

Blockwise processing of file

Description

Reads the specified file block by block and feeds each block to the specified function.

Usage

process_blocks(x, fun, ...)

## S4 method for signature 'laf'
process_blocks(
  x,
  fun,
  columns = 1:ncol(x),
  nrows = 5000,
  allow_interupt = FALSE,
  progress = FALSE,
  ...
)

Arguments

x	an object the supports the process_blocks method, such as an laf object.
fun	a function to apply to each block (see details).
...	additional parameters are passed on to fun.
columns	an integer vector with the columns that should be read in.
nrows	the (maximum) number of rows to read in one block
allow_interupt	when TRUE the function fun is expected to return a list. The second element is the result of the function. The first element should be a logical value indication whether process_blocks should continue (FALSE) or stop (TRUE). When interrupted the function is not called a last time with an empty data.frame to finalize the result.
progress	show a progress bar. Note that this triggers a calculation of the number of lines in the file which for CSV files can take some time. When numeric code is used as the style of the progress bar (see txtProgressBar).

Details

The function should accept as the first argument the next block of data. When the end of the file is reached this is an empty (zero row) data.frame. As the second argument the function should accept the output of the previous call to the function. The first time the function is called the second argument has the value NULL.

djvanderlaan/LaF documentation built on Dec. 16, 2024, 7:35 p.m.