Streamer: Enabling stream processing of large files

Large data files can be difficult to work with in R, where data generally resides in memory. This package encourages a style of programming where data is 'streamed' from disk into R via a `producer' and through a series of `consumers' that, typically reduce the original data to a manageable size. The package provides useful Producer and Consumer stream components for operations such as data input, sampling, indexing, and transformation; see package?Streamer for details.

AuthorMartin Morgan, Nishant Gopalakrishnan
Date of publicationNone
MaintainerMartin Morgan <martin.morgan@roswellpark.org>
LicenseArtistic-2.0
Version1.20.0

View on Bioconductor

Functions

close.ConnectionProducer Man page
ConnectionProducer Man page
ConnectionProducer-class Man page
ConnectionProducer-classes Man page
Consumer Man page
Consumer-class Man page
DAGParam Man page
DAGParam-class Man page
DAGParam,data.frame-method Man page
DAGParam,graphNEL-method Man page
DAGParam,matrix-method Man page
DAGParam,missing-method Man page
DAGTeam Man page
DAGTeam-class Man page
Downsample Man page
Downsample-class Man page
Function* Man page
FunctionConsumer Man page
FunctionConsumer-class Man page
FunctionProducer Man page
FunctionProducer-class Man page
FunctionProducerConsumer-classes Man page
lapply,Producer-method Man page
lapply,Stream-method Man page
length,Stream-method Man page
MulticoreParam Man page
MulticoreParam-class Man page
MulticoreTeam-class Man page
ParallelParam Man page
ParallelParam-class Man page
ParallelRegister-class Man page
plot.DAGParam Man page
plot.DAGTeam Man page
Producer Man page
Producer-class Man page
RawInput Man page
RawInput-class Man page
rawParserFactory Man page
rawReaderFactory Man page
RawToChar Man page
RawToChar-class Man page
ReadLinesProducer Man page
ReadLinesProducer-class Man page
ReadTableProducer Man page
ReadTableProducer-class Man page
Reducer Man page
Reducer-class Man page
register Man page
reset Man page
reset-methods Man page
reset,Streamer-method Man page
Rev Man page
Rev-class Man page
sapply,Producer-method Man page
sapply,Stream-method Man page
ScanProducer Man page
ScanProducer-class Man page
Seq Man page
Seq-class Man page
show,Consumer-method Man page
status Man page
status-methods Man page
status,Streamer-method Man page
Stream Man page
Stream-class Man page
Stream,Consumer-method Man page
Streamer Man page
Streamer-class Man page
Streamer-package Man page
Stream-methods Man page
[[,Stream,numeric-method Man page
Stream,Producer-method Man page
Team Man page
Team-class Man page
Team,missing-method Man page
Team,MulticoreParam-method Man page
Utility Man page
Utility-class Man page
Utility-classes Man page
yield Man page
yield-methods Man page
yield,Streamer-method Man page

Files

Streamer/DESCRIPTION
Streamer/NAMESPACE
Streamer/R
Streamer/R/AllGenerics.R Streamer/R/ConnectionProducer-classes.R Streamer/R/Consumer-class.R Streamer/R/DAGParam-class.R Streamer/R/DAGParam-methods.R Streamer/R/DAGTeam-class.R Streamer/R/Downsample-class.R Streamer/R/FunctionProducerConsumer-classes.R Streamer/R/ParallelParam-classes.R Streamer/R/Producer-class.R Streamer/R/RawInput-class.R Streamer/R/Reducer-class.R Streamer/R/Seq-class.R Streamer/R/Stream-class.R Streamer/R/Streamer-class.R Streamer/R/Team-class.R Streamer/R/Team-methods.R Streamer/R/Utility-classes.R Streamer/R/lapply-methods.R Streamer/R/plot-methods.R Streamer/R/stream-methods.R Streamer/R/zzz.R
Streamer/build
Streamer/build/vignette.rds
Streamer/inst
Streamer/inst/README-ncdf4.txt
Streamer/inst/doc
Streamer/inst/doc/Streamer.R
Streamer/inst/doc/Streamer.Rnw
Streamer/inst/doc/Streamer.pdf
Streamer/inst/extdata
Streamer/inst/extdata/NetCDFData.nc
Streamer/inst/extdata/s_1_sequence.txt
Streamer/inst/scripts
Streamer/inst/scripts/NetCDFFile.R
Streamer/inst/scripts/NetCDFInput.R
Streamer/inst/scripts/testParallel.R
Streamer/inst/unitTests
Streamer/inst/unitTests/test_ConnectionProducer.R
Streamer/inst/unitTests/test_DAGTeam.R
Streamer/inst/unitTests/test_Producer.R
Streamer/inst/unitTests/test_RawInput.R
Streamer/inst/unitTests/test_Reducer.R
Streamer/inst/unitTests/test_RevStream.R
Streamer/inst/unitTests/test_Stream.R
Streamer/inst/unitTests/test_Streamer.R
Streamer/inst/unitTests/test_Team.R
Streamer/man
Streamer/man/ConnectionProducer-classes.Rd Streamer/man/Consumer-class.Rd Streamer/man/DAGTeam-class.Rd Streamer/man/Downsample-class.Rd Streamer/man/FunctionProducerConsumer-classes.Rd Streamer/man/ParallelParam-classes.Rd Streamer/man/Producer-class.Rd Streamer/man/RawInput-class.Rd Streamer/man/Reducer-class.Rd Streamer/man/Seq-class.Rd Streamer/man/Stream-class.Rd Streamer/man/Streamer-package.Rd Streamer/man/Team-class.Rd Streamer/man/Utility-classes.Rd Streamer/man/reset.Rd Streamer/man/status.Rd Streamer/man/yield.Rd
Streamer/src
Streamer/src/R_init_Streamer.c
Streamer/src/raw_input.c
Streamer/src/raw_input.h
Streamer/tests
Streamer/tests/unit_tests.R
Streamer/vignettes
Streamer/vignettes/Streamer.Rnw
Streamer/vignettes/StreamerClasses.xmi

Questions? Problems? Suggestions? or email at ian@mutexlabs.com.

Please suggest features or report bugs with the GitHub issue tracker.

All documentation is copyright its authors; we didn't write any of that.