clusterApply: Parallelize computations using a Spark cluster

In clarkfitzg/sparklite: Lightweight frontend to run R code on an Apache Spark cluster

Description

This works by serializing x onto the worker nodes, running the computation, and finally deserializing the result.

Usage

clusterApply(cl, x, fun, ...)

Arguments

cl	cluster is a Spark connection as returned from start_shell
x	R object that can be coerced to list
fun	function to evaluate

Value

list with fun evaluated at each element of x

See Also

makeClosure, lapply, clusterApply, clusterExport, in parallel package

Examples

library(sparkapi)
sc <- start_shell(master = "local")

clusterApply(sc, 1:10, function(x) x + 2)

a <- 20
helperfunc <- function(x) sin(x)
f <- function(x) helperfunc(x) + a
# To apply f you'll need to also get it's dependencies
f <- makeClosure(f, c("a", "helperfunc"))
clusterApply(sc, 1:10, f)

clarkfitzg/sparklite documentation built on May 13, 2019, 7:35 p.m.