chunk.map: Map a function over a file by chunks

In iotools: I/O Tools for Streaming

Map a function over a file by chunks

Description

A wrapper around the core iotools functions to easily apply a function over chunks of a large file. Results can be either written to a file or returned as an internal list.

Usage

chunk.map(input, output = NULL, formatter = .default.formatter,
          FUN, key.sep = NULL, max.line = 65536L,
          max.size = 33554432L, output.sep = ",", output.nsep = "\t",
          output.keys = FALSE, skip = 0L, ...)

Arguments

input	an input connection or character vector describing a local file.
output	an optional output connection or character vector describing a local file. If NULL, the results are returned internally as a list.
formatter	a function that takes raw input and produces the input given to FUN
FUN	a user provided function to map over the chunks. The result of FUN is either wrapper in a list item, when output is NULL, or written to the output file using as.output
key.sep	optional key separator given to chunk.reader
max.line	maximum number of lines given to chunk.reader
max.size	maximum size of a block as given to read.chunk
output.sep	single character giving the field separator in the output.
output.nsep	single character giving the key separator in the output.
output.keys	logical. Whether as.output should interpret row names as keys.
skip	integer giving the number of lines to strip off the input before reading. Useful when the input contains a row a column headers
...	additional parameters to pass to FUN

Value

A list of results when output is NULL; otherwise no output is returned.

Author(s)

Taylor Arnold

iotools documentation built on May 29, 2024, 5:48 a.m.