loop_tasks: Build a job target by looping over the tasks and steps in a...

In USGS-R/scipiper: Support functions for ushering data through a scientific workflow

Build a job target by looping over the tasks and steps in a task remake file

Description

Usual behavior: Attempts all steps in a task before moving on to the next task. This is especially useful if intermediate files are created and deleted over several steps within a task, and if those files would take up too much space if intermediate files from one step of all tasks were simultaneously present. This function also provides fault tolerance, retries, and using file existence as a first check on which tasks/steps are already built (faster than the remake practice of checking file hashes, which are only checked here after all files' presence suggest that everything is built).

Important: if you want the "usual behavior" from this function, delete any task targets that are files and are known to be out of date before starting. Out-of-date files will only be checked and rerun at the very end, whereas missing files will be tried as many as num_tries times.

Usage

loop_tasks(
  task_plan,
  task_makefile,
  task_names = NULL,
  step_names = NULL,
  num_tries = 30,
  sleep_on_error = 0,
  ind_ext = getOption("scipiper.ind_ext"),
  verbose = TRUE,
  force = FALSE,
  n_cores = 1
)

Arguments

task_plan	task plan as created by create_task_plan()
task_makefile	file name of the .yml makefile for these tasks
task_names	NULL to build all tasks, or character vector of specific tasks to build
step_names	NULL to build all final_steps from the task plan (see create_task_plan), or character vector of specific steps to build
num_tries	integer number of times to retry looping through all remaining tasks
sleep_on_error	integer number of seconds to sleep immediately after a failed task. Especially useful if the error was likely to be inconsistent (e.g., a temporary network issue) and might not occur again if we wait a while
ind_ext	the indicator file extension passed to scmake, identifying those files for which build/status information will be shared via git-committable files in the build/status folder. You should git commit the resulting build/status files.
verbose	define the format of task messages. Use TRUE for progress bar for the status of each task, and FALSE for no output
force	logical. If TRUE, targets specified by task_names (NULL for all tasks) and step_names (NULL for the final step) will be deleted with scdel before being built.
n_cores	integer How many cores should be utilized when executing the task plan? Defaults to one (no parallelization).

USGS-R/scipiper documentation built on May 25, 2023, 8:47 a.m.