tar_backoff: Exponential backoff

In targets: Dynamic Function-Oriented 'Make'-Like Declarative Pipelines

Exponential backoff

Description

Configure exponential backoff while polling for tasks during the pipeline.

Usage

tar_backoff(min = 0.001, max = 0.1, rate = 1.5)

Arguments

min	Positive numeric of length 1, minimum polling interval in seconds. Must be at least sqrt(.Machine$double.eps).
max	Positive numeric of length 1, maximum polling interval in seconds. Must be at least sqrt(.Machine$double.eps).
rate	Positive numeric of length 1, greater than or equal to 1. Multiplicative rate parameter that allows the exponential backoff minimum polling interval to increase from min to max. Actual polling intervals are sampled uniformly from the current minimum to max.

Details

This function is for advanced usage only. Most users should not need to modify the default exponential backoff. To configure exponential backoff for a pipeline, supply the output of tar_backoff() to the backoff argument of tar_option_set() in the ⁠_targets.R⁠ file. See the Backoff section of the help file for details.

Backoff

In high-performance computing it can be expensive to repeatedly poll the priority queue if no targets are ready to process. The number of seconds between polls is runif(1, min, max(max, min * rate ^ index)), where index is the number of consecutive polls so far that found no targets ready to skip or run, and min, max, and rate are arguments to tar_backoff(). (If no target is ready, index goes up by 1. If a target is ready, index resets to 0. For more information on exponential, backoff, visit https://en.wikipedia.org/wiki/Exponential_backoff). Raising min or max is kinder to the CPU etc. but may incur delays in some instances.

See Also

Other utilities: tar_active(), tar_call(), tar_cancel(), tar_definition(), tar_envir(), tar_group(), tar_name(), tar_path_script_support(), tar_path_script(), tar_path_store(), tar_path_target(), tar_path(), tar_seed(), tar_source(), tar_store()

Examples

if (identical(Sys.getenv("TAR_EXAMPLES"), "true")) { # for CRAN
tar_dir({ # tar_dir() runs code from a temp dir for CRAN.
tar_option_set(backoff = tar_backoff(min = 0.001, max = 0.1, rate = 1.5))
})
}

targets documentation built on Oct. 12, 2023, 5:07 p.m.