mpi_lapply: mpi_napply

In RBigData/tasktools: Tools for Task Parallelism with MPI

Description

A distributed lapply() function.

Usage

mpi_lapply(
  X,
  FUN,
  ...,
  checkpoint_path = NULL,
  checkpoint_freq = 1,
  preschedule = TRUE
)

Arguments

X	A list or vector on rank 0 autmoatically distributed to other ranks. Values on other ranks will be ignored (passing NULL is recommended).
FUN	Function to evaluate.
...	Additional arguments passed to FUN.
checkpoint_path	If a path is specified, then each MPI rank will write checkpoints to disk during execution. If this path is global (the same on all ranks), then that path should be accessible to all ranks. However, a local path pointing to node-local storage can also be used. All checkpoint files will be removed on successful completion of the function. If the value is the default NULL, then no checkpointing takes place.
checkpoint_freq	The checkpoint frequency; a positive integer.
preschedule	Should the jobs be distributed among the MPI ranks up front? Otherwise, the jobs will be evaluated on a "first come first serve" basis among the ranks.

Details

The vector/list X should be on rank 0. If it is already distributed, then you should just cally lapply() on the (already) local data.

If preschedule=FALSE then jobs are likely to be evaluated out of order (that's actually the point). However, the return is reconstructed in the linear order, so that the first element of the return list is the value resulting from evaluating FUN at 1, the second at 2, and so on.

Value

A list on rank 0.

RBigData/tasktools documentation built on April 10, 2020, 2:36 p.m.