RawInput-class: Class "RawInput"

In Streamer: Enabling stream processing of large files

Description

A Producer-class to interpret files as raw (binary) data. Users interact with this class through the constructor RawInput and methods yield, reset, and Stream.

This class requires two helper functions; the ‘factory’ methods defined on this page can be used to supply these. rawReaderFactory creates a ‘reader’, whose responsibility it is to accept a connection and return a vector of predefined type, e.g., raw. rawParserFactory creates a ‘parser’, responsible for parsing a buffer and vector of the same type as produced by the reader into records. The final record may be incomplete (e.g., because reader does not return complete records), and regardless of completion status is the content of buf on the subsequent invocation of parser. length(buf) or length(bin) may be 0, as when the first or final record is parsed.

Usage

RawInput(con, yieldSize = 1e+06, reader = rawReaderFactory(), 
    parser = rawParserFactory(), ...)
rawReaderFactory(blockSize = 1e+06, what)
rawParserFactory(separator = charToRaw("\n"), trim = separator)

Arguments

con	A character string or connection (opened as "rb" mode) from which raw input will be retrieved.
yieldSize	The number of records the input parser is to yield.
reader	A function of one argument (con, an open connection positioned at the start of the file, or at the position the con was in at the end of the previous invocation of the reader function) that returns a vector of type raw.
parser	A function of two arguments (buf, bin), parsing the raw vector c(buf, bin) into records.
...	Additional arguments, passed to the $new method of this class. Currently ignored.

blockSize	The number of bytes to read at one time.
what	The type of data to read, as the argument to readBin.

separator	A raw vector indicating the unique sequence of bytes by which record starts are to be recognized. The parser supplied here includes the record separator at the start of each record.
trim	A raw vector that is a prefix of separator, and that is to be removed from the record during parsing.

Fields

con:

Object of class connection. An R connection opened in “rb” mode from which data will be read.

blockSize:

Object of class integer. Size (e.g., number of raw bytes) input during each yield.

reader:

Object of class function. A function used to input blockSize elements. See rawReaderFactory.

parser:

Object of class function. A function used to parse raw input into records, e.g., breaking a raw vector on new lines ‘\n’. See rawParserFactory

.buffer:

Object of class raw. Contains read but not parsed raw stream data.

.records:

Object of class list. Parsed but not yet yield-ed records.

.parsedRecords:

Object of class integer. Total number of records parsed by the Producer.

Class-Based Methods

reset():

Remove buffer and current records, reset record counter, re-open con.

Author(s)

Martin Morgan [e-mail]

See Also

Stream

Examples

fl <- system.file("extdata", "s_1_sequence.txt", package="Streamer")
b <- RawInput(fl, 100L, reader=rawReaderFactory(1e4))
length(value <- yield(b))
head(value)
close(b)

Streamer documentation built on Nov. 8, 2020, 5:53 p.m.