[Python-Dev] python subprocess module to submit a list of slurm sbatch jobs, each job use multiprocessing.Pool to run simulation on single compute node in cluster

[Tian, Liang]

Dear Developers,
I have been working on a piece of code development that need your sincere help (three code file attached here).

I use run_slurm.py to use python subprocess module to submit multiple jobs to slurm cluster by invoking a sbatch file with a for loop to reach the following target:

1) create some environmental variables for job_slurm.py to run simulations (values of these environmental variables will change for each job with the for loop)

2) invoke submit_slurm.sh to submit a sbatch job that will run job_slurm.py

3) each job_slurm.py will use python multiprocess.Pool to run parallized simulations on each ./mod_bart.sh exeutable file (will run for a few hours) on a single compute node in cluster using all cores of this compute node

I get the following error all the time, could you provide some insights on how our implementation is wrong to achieve the desired goal:


Exception in thread Thread-3:
Traceback (most recent call last):
  File "/share/apps/python/2.7/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/share/apps/python/2.7/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/share/apps/python/2.7/lib/python2.7/multiprocessing/pool.py", line 347, in _handle_results
    task = get()
TypeError: ('__init__() takes at least 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ())


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190108/7117a0e8/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: run_slurm.py
Type: text/x-python-script
Size: 2268 bytes
Desc: run_slurm.py
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190108/7117a0e8/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: submit_slurm.sh
Type: text/x-sh
Size: 133 bytes
Desc: submit_slurm.sh
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190108/7117a0e8/attachment.sh>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: job_slurm.py
Type: text/x-python-script
Size: 1055 bytes
Desc: job_slurm.py
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190108/7117a0e8/attachment-0001.bin>