In pablo-gar/rSubmitter: An interface between R and a SLURM cluster

Intro

The Job class in rSubmitter allows for submission and management of cluster jobs in a SLURM cluster from R. It does so in a semi-interactive fashion providing fleixibility to work in a SLURM cluster either from an interactive R session or from an automated R script.

This tutorial describes in full detail the capabilities of the Job class

Creating a job

The first step to submit a job is to create an object of the Job class with some relevant information regarding the commands to execute and the SLURM parameters associated to the job.

For example, if you want to submit a job with 8Gb of memory requested, you create a Job object like this:

library("rSubmitter")
bash_cmd <- "echo Hello World"
my_job <- Job$new(bash_cmd, mem = "8G")

The Job class is an R6 class, hence the use of the Job$new() constructor method to create a new object. my_job now contains all the information necessary to submit a job with 8Gb and a simple "Hello world" bash command.

Being an R6 class object, my_job contains several methods to manipulate or execute actions related to this specific instance. These methods are called using the $ operator, for example my_job$submit() submits the job associated to this object.

Before we explore the submission process let's dive into the options available for you when creating a new Job, since you won't be able to change them in the future.

Commands to execute

The main prupose of creating and submitting a job is to execute one or more commands in a cluster. When creating a job the only required argument is commandVector, which is the first positional argument too. This has to be a character vector with one or more bash commands to be executed in the cluster, for example this job will create a "Hello World" file and then will copy it:

library("rSubmitter")
bash_cmd <- c("echo Hello World > temp_file.txt", "cp temp_file.txt temp_file_copy.txt")
my_job <- Job$new(commandVector = bash_cmd)

Alternatively you can provide a single string with commands separated by ";" following the bash syntax.

Currently the commands can only be bash commands and assumes that /bin/bash is available.

Job name

You can specify a job name through the argument jobName of the $new() constructor method. This is useful as rSubmitter will use it as a prefix for output files, see below for more details.

The default job name is randomly generated with the form rSubmitter_job_[random_alphanumeric]

Requesting computing capabilities

The computing capabilities required for your job have to be requested at the time of creation. These are passed as arguments to the $new() constructor method, currently the following options are available, you can check the reference to see the most updated list of options.

• partition : character - Partition to use. Equivalent to --partition of SLURM sbatch
• time : character - Time requested for job execution, one accepted format is "HH:MM:SS". Equivalent to --time of SLURM sbatch
• mem : character - Memory requested for job execution, one accepted format is "xG" or "xMB". Equivalent to --mem of SLURM sbatch
• proc : integer - Number of processors requested per task. Equivalent to --cpus-per-task of SLURM sbatch
• totalProc : integer - Number of tasks requested for job. Equivalent to --ntasks of SLURM sbatch
• nodes : integer - Number of nodes requested for job. Equivalent to --nodes of SLURM sbatch

STDERR, STDOUT and SLURM batch files

The the standard error, output files, and sbatch script are streamed into files in the current working directory with the file name of the form jobName.[err|out|sbatch]; where jobName can be specified when creating the Job object (see above)

You can change the folder where these files will be generated through the outDir argument of the $new() constructor method.

Full exmaple

This example demonstrates the use of all avaialbe options when creating a new Job

library("rSubmitter")
bash_cmd <- c("echo Hello World", "echo Goodbye")
my_job <- Job$new(commandVector = bash_cmd,
                  jobName = "dummy_job",
                  outDir = "~",
                  partition = "normal", time = "2:00:00", mem = "8G", 
                  proc = 1, totalProc = 1, nodes = 1)

Eventually the following files will be created (after job submission)

~/dummy_job.out
~/dummy_job.err
~/dummy_job.sbatch

Submitting a job

Once you created a Job object you can submit it to SLURM using its method $submit(). This will create a sbatch script and submit it to the queue.

library("rSubmitter")
bash_cmd <- "echo Hello World"
my_job <- Job$new(bash_cmd, mem = "2G", time = "2:00")

my_job$submit()

Monitoring a job

After submission there are two main ways to monitor the status of a job. You can manually inspect the state of a job by using the method $getState() which will return a data.frame with three columns: the SLURM job id, the job name and its current state.

my_job$submit()
last_state <- my_job$getState()

last_state
#     jobId                     jobName   jobState
#1 24339250   rSubmitter_job_1324462985    PENDING

The second and more recommended way to monitor a job is to use the $wait() method. This will put R in idle until the job is finished or failed, printing the status of the job every N seconds (N can be set in the config file). For convience if the job fails $wait() will throw an error.

my_job$submit()
my_job$wait()

# 2018-08-27 15:17:25 --- Cluster Status |  PENDING = 1 |
# 2018-08-27 15:20:53 --- Cluster Status |  COMPLETED = 1 |

Cancelling a job

You can cancel any submitted jobs using their $cancel() method. This will send a kill signal to a submitted job associated to this object. After the signal has been sent SLURM can take some time before completing the request.

my_job$submit()
my_job$cancel()

# 2018-08-27 15:18:10 Cancelling 1 job(s)
# 2018-08-27 15:18:10 Finished sending cancel signal

Cleaning job-associated files

After a job is completed you may want to remove all the files created by rSubmitter that are associated to the job. The three files that are generated are STDOUT, STDERR and the sbatch script. You can delete one or more of these using the $clean() method and specifying which ones you'd like to remove. By default all three are removed.

my_job$submit()
my_job$cancel()
my_job$clean(script = TRUE, out = TRUE, err = TRUE)

$clean() will throw an error if the job is submitted and it has not completed.

pablo-gar/rSubmitter documentation built on Jan. 26, 2020, 2:08 a.m.