record: Record and replay iterators

In iterors: Fast, Compact Iterators and Tools

Record and replay iterators

Description

The record function records the values issued by a specified iterator to a file or connection object. The ireplay function returns an iterator that will replay those values. This is useful for iterating concurrently over multiple, large matrices or data frames that you can't keep in memory at the same time. These large objects can be recorded to files one at a time, and then be replayed concurrently using minimal memory.

Usage

record(iterable, con, ...)

Arguments

iterable	The iterable to record to the file.
con	A file path or open connection.
...	passed along to iteror(iterable, ...)

Details

Originally from the itertools package.

Value

NULL, invisibly.

Examples

suppressMessages(library(foreach))

m1 <- matrix(rnorm(70), 7, 10)
f1 <- tempfile()
record(iteror(m1, by='row', chunkSize=3), f1)

m2 <- matrix(1:50, 10, 5)
f2 <- tempfile()
record(iteror(m2, by='column', chunkSize=3), f2)

# Perform a simple out-of-core matrix multiply
p <- foreach(col=ireplay(f2), .combine='cbind') %:%
       foreach(row=ireplay(f1), .combine='rbind') %do% {
         row %*% col
       }

dimnames(p) <- NULL
print(p)
all.equal(p, m1 %*% m2)
unlink(c(f1, f2))

iterors documentation built on May 31, 2023, 5:36 p.m.