loop_tasks: Build a job target by looping over the tasks and steps in a...

Description Usage Arguments

View source: R/loop_tasks.R

Description

Usual behavior: Attempts all steps in a task before moving on to the next task. This is especially useful if intermediate files are created and deleted over several steps within a task, and if those files would take up too much space if intermediate files from one step of all tasks were simultaneously present. This function also provides fault tolerance, retries, and using file existence as a first check on which tasks/steps are already built (faster than the remake practice of checking file hashes, which are only checked here after all files' presence suggest that everything is built).

Important: if you want the "usual behavior" from this function, delete any task targets that are files and are known to be out of date before starting. Out-of-date files will only be checked and rerun at the very end, whereas missing files will be tried as many as num_tries times.

Usage

1
2
3
loop_tasks(task_plan, task_makefile, task_names = NULL,
  step_names = NULL, num_tries = 30, sleep_on_error = 0,
  ind_ext = getOption("scipiper.ind_ext"), verbose = TRUE)

Arguments

task_plan

task plan as created by create_task_plan()

task_makefile

file name of the .yml makefile for these tasks

task_names

NULL to build all tasks, or character vector of specific tasks to build

step_names

NULL to build all final_steps from the task plan (see create_task_plan), or character vector of specific steps to build

num_tries

integer number of times to retry looping through all remaining tasks

sleep_on_error

integer number of seconds to sleep immediately after a failed task. Especially useful if the error was likely to be inconsistent (e.g., a temporary network issue) and might not occur again if we wait a while

vebose

define the format of task messages. Use TRUE for progress bar for the status of each task, and FALSE for no output


USGS-R/scipiper documentation built on Aug. 14, 2018, 5:23 a.m.