doc/jobs_in_parallel_fail/README.md

In richelbilderbeek/peregrine: Work With The Groninger Peregrine Computer Cluster

jobs_in_parallel_fail

Goal of this is to create a pipeline as such:

job_1.sh -> file_1.txt -> job_2.sh -> job_3.sh
            file_2.txt -> job_2.sh -> 
            file_3.txt -> job_2.sh -> 
            file_4.txt -> job_2.sh -> 
            file_5.txt -> job_2.sh -> 
            etcetera 

In this pipeline: 1) a random number from one to ten files are created 2) each file must be processed in parallel 3) after all jobs have finished, these files must be processed

All jobs are started with:

jobs.sh

The idea was that jobs.sh would oversee the entire operations.

That is an incorrect idea: jobs.sh cannot predict when the slaves of job_2 have finished.

The road to success it to let job_2_master.sh start job_3.sh.

In detail

                          +-----------------+
                          | job_2_master.sh |
                          +-----------------+
job_1.sh -> file_1.txt -> | job_2_slave.sh  | -> job_3.sh
            file_2.txt -> | job_2_slave.sh  | ->
            file_3.txt -> | job_2_slave.sh  | ->
            file_4.txt -> | job_2_slave.sh  | ->
            file_5.txt -> | job_2_slave.sh  | ->
            etcetera      +-----------------+

richelbilderbeek/peregrine documentation built on Jan. 3, 2024, 7:42 p.m.