batchjobs_multicore: BatchJobs multicore futures

In future.BatchJobs: A Future API for Parallel and Distributed Processing using BatchJobs

Description

A BatchJobs multicore future is an asynchronous multiprocess future that will be evaluated in a background R session.

We highly recommend using multisession (sic!) futures of the future package instead of multicore BatchJobs futures.

Usage

batchjobs_multicore(
  expr,
  envir = parent.frame(),
  substitute = TRUE,
  globals = TRUE,
  label = "BatchJobs",
  workers = availableCores(constraints = "multicore"),
  job.delay = FALSE,
  ...
)

Arguments

expr	The R expression to be evaluated
envir	The environment in which global environment should be located.
substitute	Controls whether expr should be substitute():d or not.
globals	(optional) a logical, a character vector, a named list, or a Globals object. If TRUE, globals are identified by code inspection based on expr and tweak searching from environment envir. If FALSE, no globals are used. If a character vector, then globals are identified by lookup based their names globals searching from environment envir. If a named list or a Globals object, the globals are used as is.
label	(optional) Label of the future (where applicable, becomes the job name for most job schedulers).
workers	The number of multicore processes to be available for concurrent BatchJobs multicore futures.
job.delay	(optional) Passed as is to submitJobs().
...	Additional arguments passed to BatchJobsFuture().

Details

BatchJobs multicore futures rely on the BatchJobs backend set up by makeClusterFunctionsMulticore(). The BatchJobs multicore backend only works on operating systems supporting the ps command-line tool, e.g. Linux and OS X. However, they are not supported on neither Windows nor Solaris Unix (because ps -o ucomm= is not supported). When not supported, it falls back to batchjobs_local.

Warning: For multicore BatchJobs, the BatchJobs package uses a built-in algorithm for load balancing based on other processes running on the same machine. This is done in order to prevent the machine's CPU load to blow up. Unfortunately, the BatchJobs criteria for handling this often results in starvation, that is, long waiting times before launching jobs. The risk for this is particularly high if there are other R processes running on the same machine including those by other users. See also https://github.com/tudo-r/BatchJobs/issues/99. Conclusion: We highly recommend using multisession futures of the future package instead of multicore BatchJobs futures.

Also, despite the name, BatchJobs multicore futures are in function closer related to multisession futures than multicore futures, both provided by the future package. This is because BatchJobs spawns off background R sessions rather than forking the current R process as the name otherwise might imply (at least that is how the term "multicore processing" is typically used in the R world).

Value

An object of class BatchJobsFuture.

future.BatchJobs documentation built on Jan. 5, 2021, 1:07 a.m.