array_submit_num: Submit an array job for a given number of tasks

In LieberInstitute/sgejobs: Helper functions for SGE jobs at JHPCE

Submit an array job for a given number of tasks

Description

SGE limits the number of tasks an array job can have when you submit it. At JHPCE that limit is 75000, which means that if you want to submit an array job for 75001 tasks, you first need to submit the job for task IDs 1 till 75000, and then for 75001 by itself. This function simplifies that process for you.

Usage

array_submit_num(
  job_bash,
  array_num,
  submit =
    file.exists("/cm/shared/apps/sge/sge-8.1.9/default/common/accounting_20191007_0300.txt"),
  restore = TRUE,
  array_max = 75000L
)

task_ids_num(array_num, array_max = 75000L)

Arguments

job_bash	A character(1) vector with the name of a bash script in the current working directory.
array_num	An integer(1) number of tasks to submit for the given job. The function will deal with the case where array_num is greater than array_max.
submit	A logical(1) vector determining whether to actually submit the tasks or not using qsub.
restore	A logical(1) vector determining whether to restore the script to the original state.
array_max	A maximum number of task per SGE array job. At JHPCE, that is 75000.

Value

array_submit_num: Uses array_submit() to submit an array job for a given number of tasks as determined by task_ids_num().

task_ids_num: A character vector of SGE task IDs compliant with array_max.

Author(s)

Leonardo Collado-Torres

Examples

## Choose a script name
job_name <- paste0("array_submit_num_example_", Sys.Date())

## Create an array job on the temporary directory
with_wd(tempdir(), {
    ## Create an array job script to use for this example
    job_single(
        name = job_name,
        create_shell = TRUE,
        task_num = 1
    )

    ## Now we can submit the SGE job for a given number of tasks
    array_submit_num(
        job_bash = paste0(job_name, ".sh"),
        array_num = 75001,
        submit = FALSE
    )
})


## Get the list of task IDs for a different set of task numbers
task_ids_num(1)
task_ids_num(75000)
task_ids_num(75001)
task_ids_num(150001)

LieberInstitute/sgejobs documentation built on May 12, 2023, 5:20 p.m.