makeDockerCluster: Create a docker cluster

In DockerParallel: Using the Docker Container to Create R Workers on Local or Cloud Platform

Description

Create a docker cluster. The user needs to provide a cloud provider and a worker container to make it work.

Usage

makeDockerCluster(
  cloudProvider = NULL,
  workerContainer = NULL,
  workerNumber = 1,
  workerCpu = 1024,
  workerMemory = 2048,
  workerHardwareId = character(0),
  serverCpu = 256,
  serverMemory = 2048,
  serverHardwareId = character(0),
  jobQueueName = "DockerParallelQueue",
  privateServerData = NULL,
  serverContainer = getServerContainer(workerContainer),
  stopClusterOnExit = TRUE,
  verbose = 1
)

Arguments

cloudProvider	A CloudProvider object, the cloud that the container will be deployed
workerContainer	A DockerContainer object, the object that defines the worker container
workerNumber	Integer, the number of workers in the cluster
serverCpu, workerCpu	Integer, the CPU unit used by the server or each worker. 1024 CPU unit corresponds to a physical CPU core.
serverMemory, workerMemory	Integer, the memory used by the server or each worker in MB
serverHardwareId, workerHardwareId	Character, the ID of the hardware, this argument might be ignored by some cloud providers.
jobQueueName	Character, the job queue name used by the cluster to send the job.
privateServerData	A data object made from CloudPrivateServer. If this object is provided, the cluster server should be from another source and the cloud provider will not deploy the server container.
serverContainer	A DockerContainer object, the object that defines the server container.
stopClusterOnExit	Logical, whether to stop the cluster when the cluster has been removed from the R session. The default value is TRUE.
verbose	Integer, the verbose level

Details

This is the core function of the DockerParallel package which defines the cluster object. To user the function, you need to at least provide the cloud provider and worker container. Currently we have ECSFargateProvider and BiocFERContainer, see example.

Value

A DockerCluster object

Examples

## Not run: 
## Load the ECS fargate provider
library(ECSFargateProvider)
provider <- ECSFargateProvider()

## Load the bioconductor foreach redis container
container <- BiocFERWorkerContainer()

## Define a cluster with 2 workers,
## each worker use one fourth CPU core and 512 MB memory
cluster <- makeDockerCluster(cloudProvider = provider,
                             workerContainer = container,
                             workerNumber = 2,
                             workerCpu = 256, workerMemory = 512)
## Start the cluster
cluster$startCluster()

## rescale the worker number
cluster$setWorkerNumber(4)

## Use foreach to do the parallel computing
library(foreach)
getDoParWorkers()
foreach(x= 1:4)%dopar%{
    Sys.info()
}

## End(Not run)

DockerParallel documentation built on June 23, 2021, 5:07 p.m.