Streamer: Enabling stream processing of large files

Large data files can be difficult to work with in R, where data generally resides in memory. This package encourages a style of programming where data is 'streamed' from disk into R via a `producer' and through a series of `consumers' that, typically reduce the original data to a manageable size. The package provides useful Producer and Consumer stream components for operations such as data input, sampling, indexing, and transformation; see package?Streamer for details.

Author
Martin Morgan, Nishant Gopalakrishnan
Date of publication
None
Maintainer
Martin Morgan <martin.morgan@roswellpark.org>
License
Artistic-2.0
Version
1.20.0

View on Bioconductor

Man pages

ConnectionProducer-classes
Producer classes to read file connections
Consumer-class
Class defining methods for all consumers
DAGTeam-class
Consumer classes for directed acyclic graph evaluation
Downsample-class
Consumer class to down-sample data
FunctionProducerConsumer-classes
Classes for user-defined Producers and Consumers
ParallelParam-classes
Classes to configure parallel evaluation
Producer-class
Class defining methods for all Producers
RawInput-class
Class "RawInput"
Reducer-class
Consumer class to combine successive records
reset
Function to reset a Stream, Producer, or Consumer
Seq-class
Producer class to generate (numeric) sequences
status
Function to report current status of a stream
Stream-class
Class to represent a Producer and zero or more Consumers
Streamer-package
Package to enable stream (iterative) processing of large data
Team-class
Consumer classes for parallel evaluation
Utility-classes
Consumer classes with simple functionality, e.g., RawToChar,...
yield
Function to yield one task from a Stream or Producer

Files in this package

Streamer/DESCRIPTION
Streamer/NAMESPACE
Streamer/R
Streamer/R/AllGenerics.R
Streamer/R/ConnectionProducer-classes.R
Streamer/R/Consumer-class.R
Streamer/R/DAGParam-class.R
Streamer/R/DAGParam-methods.R
Streamer/R/DAGTeam-class.R
Streamer/R/Downsample-class.R
Streamer/R/FunctionProducerConsumer-classes.R
Streamer/R/ParallelParam-classes.R
Streamer/R/Producer-class.R
Streamer/R/RawInput-class.R
Streamer/R/Reducer-class.R
Streamer/R/Seq-class.R
Streamer/R/Stream-class.R
Streamer/R/Streamer-class.R
Streamer/R/Team-class.R
Streamer/R/Team-methods.R
Streamer/R/Utility-classes.R
Streamer/R/lapply-methods.R
Streamer/R/plot-methods.R
Streamer/R/stream-methods.R
Streamer/R/zzz.R
Streamer/build
Streamer/build/vignette.rds
Streamer/inst
Streamer/inst/README-ncdf4.txt
Streamer/inst/doc
Streamer/inst/doc/Streamer.R
Streamer/inst/doc/Streamer.Rnw
Streamer/inst/doc/Streamer.pdf
Streamer/inst/extdata
Streamer/inst/extdata/NetCDFData.nc
Streamer/inst/extdata/s_1_sequence.txt
Streamer/inst/scripts
Streamer/inst/scripts/NetCDFFile.R
Streamer/inst/scripts/NetCDFInput.R
Streamer/inst/scripts/testParallel.R
Streamer/inst/unitTests
Streamer/inst/unitTests/test_ConnectionProducer.R
Streamer/inst/unitTests/test_DAGTeam.R
Streamer/inst/unitTests/test_Producer.R
Streamer/inst/unitTests/test_RawInput.R
Streamer/inst/unitTests/test_Reducer.R
Streamer/inst/unitTests/test_RevStream.R
Streamer/inst/unitTests/test_Stream.R
Streamer/inst/unitTests/test_Streamer.R
Streamer/inst/unitTests/test_Team.R
Streamer/man
Streamer/man/ConnectionProducer-classes.Rd
Streamer/man/Consumer-class.Rd
Streamer/man/DAGTeam-class.Rd
Streamer/man/Downsample-class.Rd
Streamer/man/FunctionProducerConsumer-classes.Rd
Streamer/man/ParallelParam-classes.Rd
Streamer/man/Producer-class.Rd
Streamer/man/RawInput-class.Rd
Streamer/man/Reducer-class.Rd
Streamer/man/Seq-class.Rd
Streamer/man/Stream-class.Rd
Streamer/man/Streamer-package.Rd
Streamer/man/Team-class.Rd
Streamer/man/Utility-classes.Rd
Streamer/man/reset.Rd
Streamer/man/status.Rd
Streamer/man/yield.Rd
Streamer/src
Streamer/src/R_init_Streamer.c
Streamer/src/raw_input.c
Streamer/src/raw_input.h
Streamer/tests
Streamer/tests/unit_tests.R
Streamer/vignettes
Streamer/vignettes/Streamer.Rnw
Streamer/vignettes/StreamerClasses.xmi