cluster_utils: Cluster utitility functions

In multidplyr: A Multi-Process 'dplyr' Backend

Cluster utitility functions

Description

These functions provide useful helpers for performaning common operations. 'cluster_assign()' assigns the same value on each worker; 'cluster_assign_each()' assigns different values on each worker; 'cluster_assign_partition()' partitions vectors so that each worker gets (approximately) the same number of pieces.

Usage

cluster_assign(.cluster, ...)

cluster_assign_each(.cluster, ...)

cluster_assign_partition(.cluster, ...)

cluster_copy(cluster, names, env = caller_env())

cluster_rm(cluster, names)

cluster_library(cluster, packages)

Arguments

...	Name-value pairs
cluster, .cluster	Cluster to work on
names	Name of variables to copy.
env	Environment in which to look for varibles to copy.
packages	Character vector of packages to load

Value

Functions that modify the worker environment invisibly return 'cluster' so calls can be piped together. The other functions return lists with one element for each worker.

Examples

cl <- default_cluster()
cluster_assign(cl, a = runif(1))
cluster_call(cl, a)

# Assign different values on each cluster
cluster_assign_each(cl, b = c(1, 10))
cluster_call(cl, b)

# Partition a vector so that each worker gets approximately the
# same amount of it
cluster_assign_partition(cl, c = 1:11)
cluster_call(cl, c)

# If you want different to compute different values on each
# worker, use `cluster_call()` directly:
cluster_call(cl, d <- runif(1))
cluster_call(cl, d)

# cluster_copy() is a useful shortcut
e <- 10
cluster_copy(cl, "e")

cluster_call(cl, ls())
cluster_rm(cl, letters[1:5])
cluster_call(cl, ls())

# Use cluster_library() to load packages
cluster_call(cl, search())
cluster_library(cl, "magrittr")
cluster_call(cl, search())

multidplyr documentation built on March 31, 2023, 6:42 p.m.