loop_tasks: Build a job target by looping over the tasks and steps in a...

Description Usage Arguments

View source: R/loop_tasks.R


Usual behavior: Attempts all steps in a task before moving on to the next task. This is especially useful if intermediate files are created and deleted over several steps within a task, and if those files would take up too much space if intermediate files from one step of all tasks were simultaneously present. This function also provides fault tolerance, retries, and using file existence as a first check on which tasks/steps are already built (faster than the remake practice of checking file hashes, which are only checked here after all files' presence suggest that everything is built).

Important: if you want the "usual behavior" from this function, delete any task targets that are files and are known to be out of date before starting. Out-of-date files will only be checked and rerun at the very end, whereas missing files will be tried as many as num_tries times.


loop_tasks(task_plan, task_makefile, task_names = NULL,
  step_names = NULL, num_tries = 30, sleep_on_error = 0,
  ind_ext = getOption("scipiper.ind_ext"), verbose = TRUE)



task plan as created by create_task_plan()


file name of the .yml makefile for these tasks


NULL to build all tasks, or character vector of specific tasks to build


NULL to build all final_steps from the task plan (see create_task_plan), or character vector of specific steps to build


integer number of times to retry looping through all remaining tasks


integer number of seconds to sleep immediately after a failed task. Especially useful if the error was likely to be inconsistent (e.g., a temporary network issue) and might not occur again if we wait a while


define the format of task messages. Use TRUE for progress bar for the status of each task, and FALSE for no output

USGS-R/scipiper documentation built on Aug. 14, 2018, 5:23 a.m.