crew_class_controller: Controller class
In wlandau/crew: A Distributed Worker Launcher Framework

crew_class_controller

R Documentation

Controller class

Description

R6 class for controllers.

Details

See crew_controller().

Active bindings

client: Router object.
launcher: Launcher object.
tasks: A list of mirai::mirai() task objects.
pushed: Number of tasks pushed since the controller was started.
popped: Number of tasks popped since the controller was started.
crashes_max: See crew_controller().
backup: See crew_controller().
error: Tibble of task results (with one result per row) from the last call to ⁠map(error = "stop)⁠.
backlog: Character vector of explicitly backlogged tasks.
autoscaling: TRUE or FALSE, whether async later-based auto-scaling is currently running
queue: Queue of resolved unpopped/uncollected tasks.

Methods

Public methods

crew_class_controller$new()
crew_class_controller$validate()
crew_class_controller$empty()
crew_class_controller$nonempty()
crew_class_controller$resolved()
crew_class_controller$unresolved()
crew_class_controller$unpopped()
crew_class_controller$saturated()
crew_class_controller$start()
crew_class_controller$started()
crew_class_controller$launch()
crew_class_controller$scale()
crew_class_controller$autoscale()
crew_class_controller$descale()
crew_class_controller$crashes()
crew_class_controller$push()
crew_class_controller$walk()
crew_class_controller$map()
crew_class_controller$pop()
crew_class_controller$collect()
crew_class_controller$promise()
crew_class_controller$wait()
crew_class_controller$push_backlog()
crew_class_controller$pop_backlog()
crew_class_controller$summary()
crew_class_controller$cancel()
crew_class_controller$pids()
crew_class_controller$terminate()

Method `new()`

mirai controller constructor.

Usage

crew_class_controller$new(
  client = NULL,
  launcher = NULL,
  crashes_max = NULL,
  backup = NULL
)

Arguments

client: Router object. See crew_controller().
launcher: Launcher object. See crew_controller().
crashes_max: See crew_controller().
backup: See crew_controller().

Returns

An R6 controller object.

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
client <- crew_client()
launcher <- crew_launcher_local()
controller <- crew_controller(client = client, launcher = launcher)
controller$start()
controller$push(name = "task", command = sqrt(4))
controller$wait()
controller$pop()
controller$terminate()
}

Method `validate()`

Validate the controller.

Usage

crew_class_controller$validate()

Returns

NULL (invisibly).

Method `empty()`

Check if the controller is empty.

Usage

crew_class_controller$empty(controllers = NULL)

Arguments

controllers: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Details

A controller is empty if it has no running tasks or completed tasks waiting to be retrieved with push().

Returns

TRUE if the controller is empty, FALSE otherwise.

Method `nonempty()`

Check if the controller is nonempty.

Usage

crew_class_controller$nonempty(controllers = NULL)

Arguments

controllers: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Details

A controller is empty if it has no running tasks or completed tasks waiting to be retrieved with push().

Returns

TRUE if the controller is empty, FALSE otherwise.

Method `resolved()`

Number of resolved mirai() tasks.

Usage

crew_class_controller$resolved(controllers = NULL)

Arguments

controllers: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Details

resolved() is cumulative: it counts all the resolved tasks over the entire lifetime of the controller session.

Returns

Non-negative integer of length 1, number of resolved mirai() tasks. The return value is 0 if the condition variable does not exist (i.e. if the client is not running).

Method `unresolved()`

Number of unresolved mirai() tasks.

Usage

crew_class_controller$unresolved(controllers = NULL)

Arguments

controllers: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Returns

Non-negative integer of length 1, number of unresolved mirai() tasks.

Method `unpopped()`

Number of resolved mirai() tasks available via pop().

Usage

crew_class_controller$unpopped(controllers = NULL)

Arguments

controllers: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Returns

Non-negative integer of length 1, number of resolved mirai() tasks available via pop().

Method `saturated()`

Check if the controller is saturated.

Usage

crew_class_controller$saturated(
  collect = NULL,
  throttle = NULL,
  controller = NULL
)

Arguments

collect: Deprecated in version 0.5.0.9003 (2023-10-02). Not used.
throttle: Deprecated in version 0.5.0.9003 (2023-10-02). Not used.
controller: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Details

A controller is saturated if the number of unresolved tasks is greater than or equal to the maximum number of workers. In other words, in a saturated controller, every available worker has a task. You can still push tasks to a saturated controller, but tools that use crew such as targets may choose not to.

Returns

TRUE if the controller is saturated, FALSE otherwise.

Method `start()`

Start the controller if it is not already started.

Usage

crew_class_controller$start(controllers = NULL)

Arguments

controllers: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Details

Returns

NULL (invisibly).

Method `started()`

Check whether the controller is started.

Usage

crew_class_controller$started(controllers = NULL)

Arguments

controllers: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Details

Actually checks whether the client is started.

Returns

TRUE if the controller is started, FALSE otherwise.

Method `launch()`

Launch one or more workers.

Usage

crew_class_controller$launch(n = 1L, controllers = NULL)

Arguments

n: Number of workers to launch.
controllers: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Returns

NULL (invisibly).

Method `scale()`

Auto-scale workers out to meet the demand of tasks.

Usage

crew_class_controller$scale(throttle = TRUE, controllers = NULL)

Arguments

throttle: TRUE to skip auto-scaling if it already happened within the last seconds_interval seconds. FALSE to auto-scale every time scale() is called. Throttling avoids overburdening the mirai dispatcher and other resources.
controllers: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Details

The scale() method re-launches all inactive backlogged workers, then any additional inactive workers needed to accommodate the demand of unresolved tasks. A worker is "backlogged" if it was assigned more tasks than it has completed so far.

Methods push(), pop(), and wait() already invoke scale() if the scale argument is TRUE. For finer control of the number of workers launched, call launch() on the controller with the exact desired number of workers.

Returns

Invisibly returns TRUE if there was any relevant auto-scaling activity (new worker launches or worker connection/disconnection events) (FALSE otherwise).

Method `autoscale()`

Run worker auto-scaling in a private later loop every controller$client$seconds_interval seconds.

Usage

crew_class_controller$autoscale(controllers = NULL)

Arguments

controllers: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Details

Call controller$descale() to terminate the auto-scaling loop.

Returns

NULL (invisibly).

Method `descale()`

Terminate the auto-scaling loop started by controller$autoscale().

Usage

crew_class_controller$descale(controllers = NULL)

Arguments

controllers: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Returns

NULL (invisibly).

Method `crashes()`

Report the number of consecutive crashes of a task.

Usage

crew_class_controller$crashes(name, controllers = NULL)

Arguments

name: Character string, name of the task to check.
controllers: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Details

See the crashes_max argument of crew_controller().

Returns

Non-negative integer, number of consecutive times the task crashed.

Method `push()`

Push a task to the head of the task list.

Usage

crew_class_controller$push(
  command,
  data = list(),
  globals = list(),
  substitute = TRUE,
  seed = NULL,
  algorithm = NULL,
  packages = character(0),
  library = NULL,
  seconds_timeout = NULL,
  scale = TRUE,
  throttle = TRUE,
  name = NULL,
  save_command = NULL,
  controller = NULL
)

Arguments

command: Language object with R code to run.
data: Named list of local data objects in the evaluation environment.
globals: Named list of objects to temporarily assign to the global environment for the task. This list should include any functions you previously defined in the global environment which are required to run tasks. See the reset_globals argument of crew_controller_local().
substitute: Logical of length 1, whether to call base::substitute() on the supplied value of the command argument. If TRUE (default) then command is quoted literally as you write it, e.g. push(command = your_function_call()). If FALSE, then crew assumes command is a language object and you are passing its value, e.g. push(command = quote(your_function_call())). substitute = TRUE is appropriate for interactive use, whereas substitute = FALSE is meant for automated R programs that invoke crew controllers.
seed: Integer of length 1 with the pseudo-random number generator seed to set for the evaluation of the task. Passed to the seed argument of set.seed() if not NULL. If algorithm and seed are both NULL, then the random number generator defaults to the widely spaced worker-specific L'Ecuyer streams as supported by mirai::nextstream(). See vignette("parallel", package = "parallel") for details.
algorithm: Integer of length 1 with the pseudo-random number generator algorithm to set for the evaluation of the task. Passed to the kind argument of RNGkind() if not NULL. If algorithm and seed are both NULL, then the random number generator defaults to the recommended widely spaced worker-specific L'Ecuyer streams as supported by mirai::nextstream(). See vignette("parallel", package = "parallel") for details.
packages: Character vector of packages to load for the task.
library: Library path to load the packages. See the lib.loc argument of require().
seconds_timeout: Optional task timeout passed to the .timeout argument of mirai::mirai() (after converting to milliseconds).
scale: Logical, whether to automatically call scale() to auto-scale workers to meet the demand of the task load. Also see the throttle argument.
throttle: TRUE to skip auto-scaling if it already happened within the last seconds_interval seconds. FALSE to auto-scale every time scale() is called. Throttling avoids overburdening the mirai dispatcher and other resources.
name: Character string, name of the task. If NULL, then a random name is generated automatically. The name of the task must not conflict with the name of another task pushed to the controller. Any previous task with the same name must first be popped before a new task with that name can be pushed.
save_command: Deprecated on 2025-01-22 (crew version 0.10.2.9004) and no longer used.
controller: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Returns

Invisibly return the mirai object of the pushed task. This allows you to interact with the task directly, e.g. to create a promise object with promises::as.promise().

Method `walk()`

Apply a single command to multiple inputs, and return control to the user without waiting for any task to complete.

Usage

crew_class_controller$walk(
  command,
  iterate,
  data = list(),
  globals = list(),
  substitute = TRUE,
  seed = NULL,
  algorithm = NULL,
  packages = character(0),
  library = NULL,
  seconds_timeout = NULL,
  names = NULL,
  save_command = NULL,
  scale = TRUE,
  throttle = TRUE,
  controller = NULL
)

Arguments

command: Language object with R code to run.
iterate: Named list of vectors or lists to iterate over. For example, to run function calls f(x = 1, y = "a") and f(x = 2, y = "b"), set command to f(x, y), and set iterate to list(x = c(1, 2), y = c("a", "b")). The individual function calls are evaluated as f(x = iterate$x[[1]], y = iterate$y[[1]]) and f(x = iterate$x[[2]], y = iterate$y[[2]]). All the elements of iterate must have the same length. If there are any name conflicts between iterate and data, iterate takes precedence.
data: Named list of constant local data objects in the evaluation environment. Objects in this list are treated as single values and are held constant for each iteration of the map.
globals: Named list of constant objects to temporarily assign to the global environment for each task. This list should include any functions you previously defined in the global environment which are required to run tasks. See the reset_globals argument of crew_controller_local(). Objects in this list are treated as single values and are held constant for each iteration of the map.
substitute: Logical of length 1, whether to call base::substitute() on the supplied value of the command argument. If TRUE (default) then command is quoted literally as you write it, e.g. push(command = your_function_call()). If FALSE, then crew assumes command is a language object and you are passing its value, e.g. push(command = quote(your_function_call())). substitute = TRUE is appropriate for interactive use, whereas substitute = FALSE is meant for automated R programs that invoke crew controllers.
seed: Integer of length 1 with the pseudo-random number generator seed to set for the evaluation of the task. Passed to the seed argument of set.seed() if not NULL. If algorithm and seed are both NULL, then the random number generator defaults to the recommended widely spaced worker-specific L'Ecuyer streams as supported by mirai::nextstream(). See vignette("parallel", package = "parallel") for details.
algorithm: Integer of length 1 with the pseudo-random number generator algorithm to set for the evaluation of the task. Passed to the kind argument of RNGkind() if not NULL. If algorithm and seed are both NULL, then the random number generator defaults to the recommended widely spaced worker-specific L'Ecuyer streams as supported by mirai::nextstream(). See vignette("parallel", package = "parallel") for details.
packages: Character vector of packages to load for the task.
library: Library path to load the packages. See the lib.loc argument of require().
seconds_timeout: Optional task timeout passed to the .timeout argument of mirai::mirai() (after converting to milliseconds).
names: Optional character of length 1, name of the element of iterate with names for the tasks. If names is supplied, then iterate[[names]] must be a character vector.
save_command: Deprecated on 2025-01-22 (crew version 0.10.2.9004). The command is always saved now.
scale: Logical, whether to automatically scale workers to meet demand. See also the throttle argument.
throttle: TRUE to skip auto-scaling if it already happened within the last seconds_interval seconds. FALSE to auto-scale every time scale() is called. Throttling avoids overburdening the mirai dispatcher and other resources.
controller: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Details

In contrast to walk(), map() blocks the local R session and waits for all tasks to complete.

Returns

Invisibly returns a list of mirai task objects for the newly created tasks. The order of tasks in the list matches the order of data in the iterate argument.

Method `map()`

Apply a single command to multiple inputs, wait for all tasks to complete, and return the results of all tasks.

Usage

crew_class_controller$map(
  command,
  iterate,
  data = list(),
  globals = list(),
  substitute = TRUE,
  seed = NULL,
  algorithm = NULL,
  packages = character(0),
  library = NULL,
  seconds_interval = NULL,
  seconds_timeout = NULL,
  names = NULL,
  save_command = NULL,
  error = "stop",
  warnings = TRUE,
  verbose = interactive(),
  scale = TRUE,
  throttle = TRUE,
  controller = NULL
)

Arguments

command

Language object with R code to run.

iterate

Named list of vectors or lists to iterate over. For example, to run function calls f(x = 1, y = "a") and f(x = 2, y = "b"), set command to f(x, y), and set iterate to list(x = c(1, 2), y = c("a", "b")). The individual function calls are evaluated as f(x = iterate$x[[1]], y = iterate$y[[1]]) and f(x = iterate$x[[2]], y = iterate$y[[2]]). All the elements of iterate must have the same length. If there are any name conflicts between iterate and data, iterate takes precedence.

data

Named list of constant local data objects in the evaluation environment. Objects in this list are treated as single values and are held constant for each iteration of the map.

globals

Named list of constant objects to temporarily assign to the global environment for each task. This list should include any functions you previously defined in the global environment which are required to run tasks. See the reset_globals argument of crew_controller_local(). Objects in this list are treated as single values and are held constant for each iteration of the map.

substitute

Logical of length 1, whether to call base::substitute() on the supplied value of the command argument. If TRUE (default) then command is quoted literally as you write it, e.g. push(command = your_function_call()). If FALSE, then crew assumes command is a language object and you are passing its value, e.g. push(command = quote(your_function_call())). substitute = TRUE is appropriate for interactive use, whereas substitute = FALSE is meant for automated R programs that invoke crew controllers.

seed

Integer of length 1 with the pseudo-random number generator seed to set for the evaluation of the task. Passed to the seed argument of set.seed() if not NULL. If algorithm and seed are both NULL, then the random number generator defaults to the recommended widely spaced worker-specific L'Ecuyer streams as supported by mirai::nextstream(). See vignette("parallel", package = "parallel") for details.

algorithm

Integer of length 1 with the pseudo-random number generator algorithm to set for the evaluation of the task. Passed to the kind argument of RNGkind() if not NULL. If algorithm and seed are both NULL, then the random number generator defaults to the recommended widely spaced worker-specific L'Ecuyer streams as supported by mirai::nextstream(). See vignette("parallel", package = "parallel") for details.

packages

Character vector of packages to load for the task.

library

Library path to load the packages. See the lib.loc argument of require().

seconds_interval

Deprecated on 2025-01-17 (crew version 0.10.2.9003). Instead, the seconds_interval argument passed to crew_controller_group() is used as seconds_max in a crew_throttle() object which orchestrates exponential backoff.

seconds_timeout

Optional task timeout passed to the .timeout argument of mirai::mirai() (after converting to milliseconds).

names

Optional character string, name of the element of iterate with names for the tasks. If names is supplied, then iterate[[names]] must be a character vector.

save_command

Deprecated on 2025-01-22 (crew version 0.10.2.9004). The command is always saved now.

error

Character of length 1, choice of action if a task was not successful. Possible values:

"stop": throw an error in the main R session instead of returning a value. In case of an error, the results from the last errored map() are in the error field of the controller, e.g. controller_object$error. To reduce memory consumption, set controller_object$error <- NULL after you are finished troubleshooting.
"warn": throw a warning. This allows the return value with all the error messages and tracebacks to be generated.
"silent": do nothing special. NOTE: the only kinds of errors considered here are errors at the R level. A crashed tasks will return a status of "crash" in the output and not trigger an error in map() unless crashes_max is reached.

warnings

Logical of length 1, whether to throw a warning in the interactive session if at least one task encounters an error.

verbose

Logical of length 1, whether to print progress messages.

scale

Logical, whether to automatically scale workers to meet demand. See also the throttle argument.

throttle

TRUE to skip auto-scaling if it already happened within the last seconds_interval seconds. FALSE to auto-scale every time scale() is called. Throttling avoids overburdening the mirai dispatcher and other resources.

controller

Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Details

map() cannot be used unless all prior tasks are completed and popped. You may need to wait and then pop them manually. Alternatively, you can start over: either call terminate() on the current controller object to reset it, or create a new controller object entirely.

Returns

A tibble of results and metadata: one row per task and columns corresponding to the output of pop().

Method `pop()`

Pop a completed task from the results data frame.

Usage

crew_class_controller$pop(
  scale = TRUE,
  collect = NULL,
  throttle = TRUE,
  error = NULL,
  controllers = NULL
)

Arguments

scale

Logical of length 1, whether to automatically call scale() to auto-scale workers to meet the demand of the task load. Scaling up on pop() may be important for transient or nearly transient workers that tend to drop off quickly after doing little work. See also the throttle argument.

collect

Deprecated in version 0.5.0.9003 (2023-10-02).

throttle

error

NULL or character of length 1, choice of action if the popped task threw an error. Possible values:

"stop": throw an error in the main R session instead of returning a value.
"warn": throw a warning.
NULL or "silent": do not react to errors. NOTE: the only kinds of errors considered here are errors at the R level. A crashed tasks will return a status of "crash" in the output and not trigger an error in pop() unless crashes_max is reached.

controllers

Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Details

If not task is currently completed, pop() will attempt to auto-scale workers as needed.

Returns

If there is no task to collect, return NULL. Otherwise, return a one-row tibble with the following columns.

name: the task name.
command: a character string with the R command.
result: a list containing the return value of the R command. NA if the task failed.
status: a character string. "success" if the task succeeded, "cancel" if the task was canceled with the cancel() controller method, "crash" if the worker running the task exited before it could complete the task, or "error" for any other kind of error.
error: the first 2048 characters of the error message if the task status is not "success", NA otherwise. Messages for crashes and cancellations are captured here alongside ordinary R-level errors.
code: an integer code denoting the specific exit status: 0 for successful tasks, -1 for tasks with an error in the R command of the task, and another positive integer with an NNG status code if there is an error at the NNG/nanonext level. nanonext::nng_error() can interpret these codes.
trace: the first 2048 characters of the text of the traceback if the task threw an error, NA otherwise.
warnings: the first 2048 characters. of the text of warning messages that the task may have generated, NA otherwise.
seconds: number of seconds that the task ran.
seed: the single integer originally supplied to push(), NA otherwise. The pseudo-random number generator state just prior to the task can be restored using set.seed(seed = seed, kind = algorithm), where seed and algorithm are part of this output.
algorithm: name of the pseudo-random number generator algorithm originally supplied to push(), NA otherwise. The pseudo-random number generator state just prior to the task can be restored using set.seed(seed = seed, kind = algorithm), where seed and algorithm are part of this output.
controller: name of the crew controller where the task ran.
worker: name of the crew worker that ran the task.

Method `collect()`

Pop all available task results and return them in a tidy tibble.

Usage

crew_class_controller$collect(
  scale = TRUE,
  throttle = TRUE,
  error = NULL,
  controllers = NULL
)

Arguments

scale: Logical of length 1, whether to automatically call scale() to auto-scale workers to meet the demand of the task load.
throttle: TRUE to skip auto-scaling if it already happened within the last seconds_interval seconds. FALSE to auto-scale every time scale() is called. Throttling avoids overburdening the mirai dispatcher and other resources.
error: NULL or character of length 1, choice of action if the popped task threw an error. Possible values: * "stop": throw an error in the main R session instead of returning a value. * "warn": throw a warning. * NULL or "silent": do not react to errors. NOTE: the only kinds of errors considered here are errors at the R level. A crashed tasks will return a status of "crash" in the output and not trigger an error in collect() unless crashes_max is reached.
controllers: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Returns

A tibble of results and metadata of all resolved tasks, with one row per task. Returns NULL if there are no tasks to collect. See pop() for details on the columns of the returned tibble.

Method `promise()`

Create a promises::promise() object to asynchronously pop or collect one or more tasks.

Usage

crew_class_controller$promise(
  mode = "one",
  seconds_interval = 1,
  scale = NULL,
  throttle = NULL,
  controllers = NULL
)

Arguments

mode

Character of length 1, what kind of promise to create. mode must be "one" or "all". Details:

If mode is "one", then the promise is fulfilled (or rejected) when at least one task is resolved and available to pop(). When that happens, pop() runs asynchronously, pops a result off the task list, and returns a value. If the task succeeded, then the promise is fulfilled and its value is the result of pop() (a one-row tibble with the result and metadata). If the task threw an error, the error message of the task is forwarded to any error callbacks registered with the promise.
If mode is "all", then the promise is fulfilled (or rejected) when there are no unresolved tasks left in the controller. (Be careful: this condition is trivially met in the moment if the controller is empty and you have not submitted any tasks, so it is best to create this kind of promise only after you submit tasks.) When there are no unresolved tasks left, collect() runs asynchronously, pops all available results off the task list, and returns a value. If the task succeeded, then the promise is fulfilled and its value is the result of collect() (a tibble with one row per task result). If any of the tasks threw an error, then the first error message detected is forwarded to any error callbacks registered with the promise.

seconds_interval

Positive numeric of length 1, delay in the later::later() polling interval to asynchronously check if the promise can be resolved.

scale

Deprecated on 2024-04-10 (version 0.9.1.9003) and no longer used. Now, promise() always turns on auto-scaling in a private later loop (if not already activated).

throttle

Deprecated on 2024-04-10 (version 0.9.1.9003) and no longer used. Now, promise() always turns on auto-scaling in a private later loop (if not already activated).

controllers

Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Details

Please be aware that pop() or collect() will happen asynchronously at a some unpredictable time after the promise object is created, even if your local R process appears to be doing something completely different. This behavior is highly desirable in a Shiny reactive context, but please be careful as it may be surprising in other situations.

Returns

A promises::promise() object whose eventual value will be a tibble with results from one or more popped tasks. If mode = "one", only one task is popped and returned (one row). If mode = "all", then all the tasks are returned in a tibble with one row per task (or NULL is returned if there are no tasks to pop).

Method `wait()`

Wait for tasks.

Usage

crew_class_controller$wait(
  mode = "all",
  seconds_interval = NULL,
  seconds_timeout = Inf,
  scale = TRUE,
  throttle = TRUE,
  controllers = NULL
)

Arguments

mode: Character of length 1: "all" to wait for all tasks to complete, "one" to wait for a single task to complete.
seconds_interval: Deprecated on 2025-01-17 (crew version 0.10.2.9003). Instead, the seconds_interval argument passed to crew_controller_group() is used as seconds_max in a crew_throttle() object which orchestrates exponential backoff.
seconds_timeout: Timeout length in seconds waiting for tasks.
scale: Logical, whether to automatically call scale() to auto-scale workers to meet the demand of the task load. See also the throttle argument.
throttle: TRUE to skip auto-scaling if it already happened within the last seconds_interval seconds. FALSE to auto-scale every time scale() is called. Throttling avoids overburdening the mirai dispatcher and other resources.
controllers: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Details

The wait() method blocks the calling R session and repeatedly auto-scales workers for tasks that need them. The function runs until it either times out or the condition in mode is met.

Returns

A logical of length 1, invisibly. TRUE if the condition in mode was met, FALSE otherwise.

Method `push_backlog()`

Push the name of a task to the backlog.

Usage

crew_class_controller$push_backlog(name, controller = NULL)

Arguments

name: Character of length 1 with the task name to push to the backlog.
controller: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Details

pop_backlog() pops the tasks that can be pushed without saturating the controller.

Returns

NULL (invisibly).

Method `pop_backlog()`

Pop the task names from the head of the backlog which can be pushed without saturating the controller.

Usage

crew_class_controller$pop_backlog(controllers = NULL)

Arguments

controllers: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Returns

Character vector of task names which can be pushed to the controller without saturating it. If the controller is saturated, character(0L) is returned.

Method `summary()`

Summarize the workers and tasks of the controller.

Usage

crew_class_controller$summary(controllers = NULL)

Arguments

controllers: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Returns

A data frame of summary statistics on the tasks that ran on a worker and then were returned by pop() or collect(). It has one row and the following columns:

controller: name of the controller.
tasks: number of tasks.
seconds: total number of runtime in seconds.
success: total number of successful tasks.
error: total number of tasks with errors, either in the R code of the task or an NNG-level error that is not a cancellation or crash.
crash: total number of crashed tasks (where the worker exited unexpectedly while it was running the task).
cancel: total number of tasks interrupted with the cancel() controller method.
warnings: total number of tasks with one or more warnings.

Method `cancel()`

Cancel one or more tasks.

Usage

crew_class_controller$cancel(names = character(0L), all = FALSE)

Arguments

names: Character vector of names of tasks to cancel. Those names must have been manually supplied by push().
all: TRUE to cancel all tasks, FALSE otherwise. all = TRUE supersedes the names argument.

Method `pids()`

Get the process IDs of the local process and the mirai dispatcher (if started).

Usage

crew_class_controller$pids(controllers = NULL)

Arguments

controllers: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Returns

An integer vector of process IDs of the local process and the mirai dispatcher (if started).

Method `terminate()`

Terminate the workers and the mirai client.

Usage

crew_class_controller$terminate(controllers = NULL)

Arguments

controllers: Not used. Included to ensure the signature is compatible with the analogous method of controller groups.

Returns

NULL (invisibly).

Examples

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
client <- crew_client()
launcher <- crew_launcher_local()
controller <- crew_controller(client = client, launcher = launcher)
controller$start()
controller$push(name = "task", command = sqrt(4))
controller$wait()
controller$pop()
controller$terminate()
}

## ------------------------------------------------
## Method `crew_class_controller$new`
## ------------------------------------------------

if (identical(Sys.getenv("CREW_EXAMPLES"), "true")) {
client <- crew_client()
launcher <- crew_launcher_local()
controller <- crew_controller(client = client, launcher = launcher)
controller$start()
controller$push(name = "task", command = sqrt(4))
controller$wait()
controller$pop()
controller$terminate()
}

wlandau/crew documentation built on Feb. 8, 2025, 10:12 a.m.

wlandau/crew index

README.md

rdrr.io home R language documentation Run R code online

CRAN packages Bioconductor packages R-Forge packages GitHub packages

Note that we can't provide technical support on individual packages. You should contact the package authors for that.

wlandau/crew A Distributed Worker Launcher Framework

crew_class_controller: Controller class In wlandau/crew: A Distributed Worker Launcher Framework

Controller class

Description

Details

Active bindings

Methods

Public methods

Method new()

Usage

Arguments

Returns

Examples

Method validate()

Usage

Returns

Method empty()

Usage

Arguments

Details

Returns

Method nonempty()

Usage

Arguments

Details

Returns

Method resolved()

Usage

Arguments

Details

Returns

Method unresolved()

Usage

Arguments

Returns

Method unpopped()

Usage

Arguments

Returns

Method saturated()

Usage

Arguments

Details

Returns

Method start()

Usage

Arguments

Details

Returns

Method started()

Usage

Arguments

Details

Returns

Method launch()

Usage

Arguments

Returns

Method scale()

Usage

Arguments

Details

Returns

Method autoscale()

Usage

Arguments

Details

Returns

Method descale()

Usage

Arguments

Returns

Method crashes()

Usage

Arguments

Details

Returns

Method push()

Usage

Arguments

wlandau/crew
A Distributed Worker Launcher Framework

crew_class_controller: Controller class
In wlandau/crew: A Distributed Worker Launcher Framework

Method `new()`

Method `validate()`

Method `empty()`

Method `nonempty()`

Method `resolved()`

Method `unresolved()`

Method `unpopped()`

Method `saturated()`

Method `start()`

Method `started()`

Method `launch()`

Method `scale()`

Method `autoscale()`

Method `descale()`

Method `crashes()`

Method `push()`

Method `walk()`

Method `map()`

Method `pop()`

Method `collect()`

Method `promise()`

Method `wait()`

Method `push_backlog()`

Method `pop_backlog()`

Method `summary()`

Method `cancel()`

Method `pids()`

Method `terminate()`