shard-package: shard: Deterministic, Zero-Copy Parallel Execution for R

In shard: Deterministic, Zero-Copy Parallel Execution for R

shard: Deterministic, Zero-Copy Parallel Execution for R

Description

Provides a parallel execution runtime for R that emphasizes deterministic memory behavior and efficient handling of large shared inputs. 'shard' enables zero-copy parallel reads via shared/memory-mapped segments, encourages explicit output buffers to avoid large result aggregation, and supervises worker processes to mitigate memory drift via controlled recycling. Diagnostics report peak memory usage, end-of-run memory return, and hidden copy/materialization events to support reproducible performance benchmarking.

Core API

• shard_map() - Primary parallel execution entry point

• shards() - Create shard descriptors with autotuning

• results() - Extract results from a shard_map run

• succeeded() - Check if shard_map completed without failures

Zero-Copy Shared Data

• share() - Share an R object for parallel access

• fetch() - Retrieve data from a shared object

• materialize() - Alias for fetch()

• is_shared() - Check if an object is shared

• shared_info() - Get information about a shared object

Output Buffers

• buffer() - Create typed writable output buffer

• buffer_open() - Open existing buffer from another process

• buffer_path() - Get buffer path for cross-process sharing

• buffer_info() - Get buffer information

• buffer_close() - Close and release buffer

Worker Pool Management

• pool_create() - Create a supervised worker pool

• pool_stop() - Stop the worker pool

• pool_status() - Check worker status and RSS

• pool_health_check() - Monitor and recycle workers

Task Dispatch

• dispatch_chunks() - Execute chunks with supervision

• pool_lapply() - Parallel lapply with supervision

• pool_sapply() - Parallel sapply with supervision

Author(s)

Maintainer: Bradley Buchsbaum [e-mail] [copyright holder]

See Also

Useful links:

• https://bbuchsbaum.github.io/shard/

• https://github.com/bbuchsbaum/shard

• Report bugs at https://github.com/bbuchsbaum/shard/issues

shard documentation built on April 3, 2026, 9:08 a.m.