run_pipe_v2: Run automated Pipelines

In sahilseth/flowr: Streamlining Design and Deployment of Complex Workflows

Run automated Pipelines

Description

Run complete pipelines, by wrapping several steps into one convenient function.

NOTE: please use flowr version 0.9.8.9010 or higher.

In summary, this function performs the following steps:

• the argument x defines the name of the pipeline. Say, for example sleep_pipe.

• fetch_pipes: finds the pipeline definition (sleep_pipe.R, sleep_pipe.def and sleep_pipe.conf files)

• sleep_pipe(...): Create all the required commands (flowmat)

• to_flow: Use flowmat and sleep_pipe.def to create a flow object.

• submit_flow: Submit the flow to the cluster.

Usage

run_pipe_v2(
  pipe_func,
  pipe_src,
  flow_def,
  flow_conf,
  flowname,
  platform,
  flow_run_path = opts_flow$get("flow_run_path"),
  rerun_wd,
  start_from,
  execute = FALSE,
  ...
)

Arguments

pipe_func	name of the pipeline function in 'pipe_src'
pipe_src	path to pipeline script
flow_def	flow definition file
flow_conf	flow conf file with various parameters used by the flow
flowname	name for the flow for submission.
platform	what platform to use, overrides flowdef
flow_run_path	passed onto to_flow. Default it picked up from flowr.conf. Typically this is ~/flowr/runs
rerun_wd	if you need to re-run, supply the previous working dir
start_from	the step to start a rerun from. Intuitively, this is ignored in a fresh run and only used in re-running a pipeline.
execute	TRUE/FALSE
...	passed onto the pipeline function as specified in x

sahilseth/flowr documentation built on Feb. 17, 2025, 11:57 a.m.