easypar
can be used to generate scripts that submit array jobs to the LSF cluster system.
First, write in R your code.
# Computation function FUN = function(x, y){ ... } # Input for 25 array jobs that match FUN arguments PARAMS = data.frame(x = runif(25), y = runif(25)) # Generates submission files run_lsf(FUN, PARAMS)
Then, in your terminal.
# Test if the generated script runs with the first input head -1 EASYPAR_LSF_input_jobarray.csv | Rscript EASYPAR_LSF_Run.R # Submit array jobs (after loading the cluster module) bsub < EASYPAR_LSF_submission.sh
FUN
(e.g.,: FUN = ls
) that can run as a stand-alone R application, with its own parameters and that manages explicitly its dependencies. Note:
FUN
will be run as an indipendent process.
PARAMS
where every row is an input for FUN
. The column order must match FUN
arguments.Conceptually, you set up the data as for an apply(FUN, MARGIN = 1)
by row.
The input should have column names without dots or spaces; these will match the arguments of FUN
. So, for instance, an input with 2-columns will only work if FUN
has 2 parameters.
run_lsf
generates 3 files:
an R script wrapping the definition of FUN
, with extra code to call FUN
using parameters from the command line.
Your function in this script is called with a fake name;
a csv
file containing the input PARAMS
, without any header (column names), and row names.
a LSF array job submission script with N
jobs where N
are the rows of PARAMS
.
Before submitting the job, test the computation as explained above.
Cluster-specific BSUB instructions can be specified, as well as other dependencies from modules available on the cluster.
Function run_lsf
allows to:
specify a list of modules that will be added as dependencies of the LSF submission script. For instance, modules = 'R/3.5.0'
will generate the dependecy for a specific R version (3.5.0
).
customize the BSUB parameters of the generated script.
The package comes with a default BSUB configuration, that has to be updated according to your cluster setup.
library(easypar) # Default parameters in the package default_BSUB_config()
These are classical BSUB parameters:
-J
, the job ID;-P
, the project ID;-q
, the queue ID;-n
and -R
, the resources to allocate;-W
the wall time of the jobs;-o
and -e
, the outputs and the error filenames. Notice that by default
we have the job array ID in the filename, so to have one log per job.It is required to modify the default values of
-P
and-q
, the project and queue ID, according to your LSF configuration. Otherwise, the submission script will generate an error becaue the default values do not mean anything.
Modifications are done to the default list of parameters; other BSUB flags can be
used as well. No checkings on their correctness are done by easypar
.
custom_BSUB = default_BSUB_config() # More informative job ID custom_BSUB$`-J` = "bwa_aligner" # A token for a project allowed to run on the cluster custom_BSUB$`-P` = "DKSMWOP331" # A queue name that is valid on the cluster custom_BSUB$`-q` = "bioinformatics" print(custom_BSUB) # Shorter version custom_BSUB = default_BSUB_config(J = 'bwa_aligner', P = 'DKSMWOP331', q = 'bioinformatics') print(custom_BSUB)
Once the BSUB has been customized, you can either:
system
call. By default (run = FALSE
) the run_lsf
function outputs the shell command that should be used to submit the jobs, but leaves the user to submit the job. This is because we experienced some command line issues calling
modules with a system call.
An example computation follows.
# A simple function that prints some outputs FUN = function(x, y){ print(x, y) } # input for 25 array jobs PARAMS = data.frame( x = runif(25), y = runif(25) ) # generates the input files, adding some module dependencies run_lsf(FUN, PARAMS, BSUB_config = custom_BSUB, modules = 'R/3.5.0' )
If you do not try to run it automatically, command bsub < EASYPAR_LSF_submission.sh
will submit the jobs to the cluster.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.