stream_map: Stream over row-groups/datasets and map

In shard: Deterministic, Zero-Copy Parallel Execution for R

Stream over row-groups/datasets and map

Description

Applies f() to each partition and returns the list of per-partition results. This is still much cheaper than collecting the full dataset when f() returns a small summary per partition.

Usage

stream_map(x, f, ...)

## S3 method for class 'shard_row_groups'
stream_map(x, f, ...)

## S3 method for class 'shard_dataset'
stream_map(x, f, ...)

Arguments

x	A shard_row_groups or shard_dataset handle.
f	Function (chunk, ...) -> value.
...	Passed to f().

Value

A list of per-partition values, one element per row-group file.

Examples

s <- schema(x = float64())
sink <- table_sink(s, mode = "row_groups")
table_write(sink, 1L, data.frame(x = rnorm(5)))
rg <- table_finalize(sink)
nrows <- stream_map(rg, nrow)

shard documentation built on April 3, 2026, 9:08 a.m.