create_task_plan: Define a set of tasks and steps within a single job

In USGS-R/scipiper: Support functions for ushering data through a scientific workflow

Define a set of tasks and steps within a single job

Description

Create a structured list that organizes tasks and their sub-tasks (steps) for a large number of near-identical tasks

Usage

create_task_plan(
  task_names,
  task_steps,
  final_steps = sapply(task_steps, `[[`, "step_name"),
  ind_dir = NULL,
  add_complete = TRUE,
  ind_ext = getOption("scipiper.ind_ext")
)

Arguments

task_names	character vector of IDs for sets of work coordinates, with length equal to the number of tasks. You will usually use these character strings within the names for each step in each task. These names should therefore describe the aspect/aspects of a task that position it within the larger parameter space of work to be done. Examples: site IDs, model names, or character code identifying a unique row in a configuration file or data.frame.
task_steps	list of definitions steps to perform within each task. Each step definition should be created by create_task_step()
final_steps	vector of step indices (either step_names or integers) identifying those task_steps that must be completed to consider the entire task complete. If one step depends on all previous steps (as indicated by its remake ⁠depends:⁠ and ⁠command:⁠ fields) then only that one step need be included in final_steps.
ind_dir	directory path, only used within steps if add_complete==TRUE, specifying the location where task-specific indicator files should be written. Such files will always be named after their associated tasks, with '.ind' as a suffix. This path information is also passed on to create_task_makefile() so that you don't need to specify it in both places.
add_complete	logical. if TRUE, ind_dir must be supplied and a final step named 'complete' will be added to the step list. This step will depend on all final_steps and will write an indicator file with a timestamp when all final steps are complete.
ind_ext	the indicator file extension to use in creating indicator files for any complete steps (only used if add_complete==TRUE)

Value

a structured list that can be passed to create_task_makefile or create_task_table

Examples

task_config <- data.frame(
  id=c('AZ','CO','CA'),
  capital=c('Phoenix','Denver','Sacramento')
)
step1 <- create_task_step(
  step_name = 'prep',
  target_name = function(task_name, step_name, ...) {
    sprintf('%s_%s', task_name, step_name)
  },
  depends = c('A','B'),
  command = "process(target_name, I('C'))"
)
step2 <- create_task_step(
  step_name = 'plot',
  command = function(target_name, task_name, ...) {
    capital <- task_config[task_config$id == task_name, 'capital']
    sprintf('visualize(\'%s\', \'%s\')', task_name, capital)
  }
)
step3 <- create_task_step('report')
task_plan <- create_task_plan(c('AZ','CA','CO'), list(step1, step2, step3),
  final_steps='report', ind_dir='states/log')

USGS-R/scipiper documentation built on May 25, 2023, 8:47 a.m.