registerDoSpark: Register a Parallel Backend

In sparklyr: R Interface to Apache Spark

Register a Parallel Backend

Description

Registers a parallel backend using the foreach package.

Usage

registerDoSpark(spark_conn, parallelism = NULL, ...)

Arguments

spark_conn	Spark connection to use
parallelism	Level of parallelism to use for task execution (if unspecified, then it will take the value of 'SparkContext.defaultParallelism()' which by default is the number of cores available to the 'sparklyr' application)
...	additional options for sparklyr parallel backend (currently only the only valid option is 'nocompile')

Value

None

Examples

## Not run: 

sc <- spark_connect(master = "local")
registerDoSpark(sc, nocompile = FALSE)

## End(Not run)

sparklyr documentation built on Nov. 2, 2023, 5:09 p.m.