parallel_start: Start parallel clusters using 'parallel' package

View source: R/utils-control-par.R

parallel_startR Documentation

Start parallel clusters using parallel package

Description

Start parallel clusters using parallel package

Usage

parallel_start(..., .method = c("parallel", "spark"))

parallel_stop()

Arguments

...

Parameters passed to underlying functions (See Details Section)

.method

The method to create the parallel backend. Supports:

  • "parallel" - Uses the parallel and doParallel packages

  • "spark" - Uses the sparklyr package

Parallel (.method = "parallel")

Performs 3 Steps:

  1. Makes clusters using parallel::makeCluster(...). The parallel_start(...) are passed to parallel::makeCluster(...).

  2. Registers clusters using doParallel::registerDoParallel().

  3. Adds .libPaths() using parallel::clusterCall().

Spark (.method = "spark")

  • Important, make sure to create a spark connection using sparklyr::spark_connect().

  • Pass the connection object as the first argument. For example, parallel_start(sc, .method = "spark").

  • The parallel_start(...) are passed to sparklyr::registerDoSpark(...).

Examples


# Starts 2 clusters
parallel_start(2)

# Returns to sequential processing
parallel_stop()




modeltime documentation built on June 8, 2022, 1:07 a.m.