job: Describing details of the job object

In flowr: Streamlining Design and Deployment of Complex Workflows

Description

Internal function (used by to_flow), which aids in creating a job object.

Usage

job(
  cmds = "",
  name = "myjob",
  q_obj = new("queue"),
  previous_job = "",
  cpu = 1,
  memory,
  walltime,
  submission_type = c("scatter", "serial"),
  dependency_type = c("none", "gather", "serial", "burst"),
  ...
)

Arguments

cmds	the commands to run
name	name of the job
q_obj	queue object
previous_job	character vector of previous job. If this is the first job, one can leave this empty, NA, NULL, '.', or ”. In future this could specify multiple previous jobs.
cpu	no of cpu's reserved
memory	The amount of memory reserved. Units depend on the platform used to process jobs
walltime	The amount of time reserved for this job. Format is unique to a platform. Typically it looks like 12:00 (12 hours reserved, say in LSF), in Torque etc. we often see measuring in seconds: 12:00:00
submission_type	submission type: A character with values: scatter, serial. Scatter means all the 'cmds' would be run in parallel as separate jobs. Serial, they would combined into a single job and run one-by-one.
dependency_type	dependency type. One of none, gather, serial, burst. If previous_job is specified, then this would not be 'none'. [Required]
...	other passed onto object creation. Example: memory, walltime, cpu

Examples

qobj <- queue(platform="torque")

## torque job with 1 CPU running command 'sleep 2'
jobj <- job(q_obj=qobj, cmd = "sleep 2", cpu=1)

## multiple commands
cmds = rep("sleep 5", 10)

## run the 10 commands in parallel
jobj1 <- job(q_obj=qobj, cmd = cmds, submission_type = "scatter", name = "job1")

## run the 10 commands sequentially, but WAIT for the previous job to complete
jobj2 <- job(q_obj=qobj, cmd = cmds, submission_type = "serial",
   dependency_type = "gather", previous_job = "job1")

fobj <- flow(jobs = list(jobj1, jobj2))

## plot the flow
plot_flow(fobj)
## Not run: 
## dry run, only create the structure without submitting jobs
submit_flow(fobj)

## execute the jobs: ONLY works on computing cluster, would fail otherwise
submit_flow(fobj, execute = TRUE)


## End(Not run)

flowr documentation built on March 3, 2021, 1:12 a.m.