Newbie help for using multiprocessing and subprocess packages for creating child processes

[Rob Newman]

Hi All,

I am new to Python, and have a very specific task to accomplish. I  
have a command line shell script that takes two arguments:

create_graphs.sh -v --sta=STANAME

where STANAME is a string 4 characters long.

create_graphs creates a series of graphs using Matlab (among other 3rd  
party packages).

Right now I can run this happily by hand, but I have to manually  
execute the command for each STANAME. What I want is to have a Python  
script that I pass a list of STANAMEs to, and it acts like a daemon  
and spawns as many child processes as there are processors on my  
server (64), until it goes through all the STANAMES (about 200).

I posted a message on Stack Overflow (ref: http://stackoverflow.com/questions/884650/python-spawn-parallel-child-processes-on-a-multi-processor-system-use-multipro) 
  and was recommended to use the multiprocessing and subprocess  
packages. In the Stack Overflow answers, it was suggested that I use  
the process pool class in multiprocessing. However, the server I have  
to use is a Sun Sparc (T5220, Sun OS 5.10) and there is a known issue  
with sem_open() (ref: http://bugs.python.org/issue3770), so it appears  
I cannot use the process pool class.

So, below is my script (controller.py) that I have attempted to use as  
a test, that just calls the 'ls' command on a file I know exists  
rather than firing off my shell script (which takes ~ 10 mins to run  
per STANAME):

#!/path/to/python

import sys
import os
import json
import multiprocessing
import subprocess

def work(verbose,staname):
   print 'function:',staname
   print 'parent process:', os.getppid()
   print 'process id:', os.getpid()
   print "ls /path/to/file/"+staname+"_info.pf"
   # cmd will eventually get replaced with the shell script with the  
verbose and staname options
   cmd = [ "ls /path/to/file/"+staname+"_info.pf" ]
   return subprocess.call(cmd, shell=False)

if __name__ == '__main__':

   report_sta_list = ['B10A','B11A','BNLO']

   # Print out the complete station list for testing
   print report_sta_list

   # Get the number of processors available
   num_processes = multiprocessing.cpu_count()

   print 'Number of processes: %s' % (num_processes)

   print 'Now trying to assign all the processors'

   threads = []

   len_stas = len(report_sta_list)

   print "+++ Number of stations to process: %s" % (len_stas)

   # run until all the threads are done, and there is no data left
   while len(threads) < len(report_sta_list):

     # if we aren't using all the processors AND there is still data  
left to
     # compute, then spawn another thread

     print "+++ Starting to set off all child processes"

     if( len(threads) < num_processes ):

       this_sta = report_sta_list.pop()

       print "+++ Station is %s" % (this_sta)

       p = multiprocessing.Process(target=work,args=['v',this_sta])

       p.start()

       print p, p.is_alive()

       threads.append(p)

     else:

       for thread in threads:

         if not thread.is_alive():

           threads.remove(thread)

However, I seem to be running into a whole series of errors:

myhost{rt}62% controller.py
['B10A', 'B11A', 'BNLO']
Number of processes: 64
Now trying to assign all the processors
+++ Number of stations to process: 3
+++ Starting to set off all child processes
+++ Station is BNLO
<Process(Process-1, started)> True
+++ Starting to set off all child processes
+++ Station is B11A
function: BNLO
parent process: 22341
process id: 22354
ls /path/to/file/BNLO_info.pf
<Process(Process-2, started)> True
function: B11A
parent process: 22341
process id: 22355
ls /path/to/file/B11A_info.pf
Process Process-1:
Traceback (most recent call last):
   File "/opt/csw/lib/python/multiprocessing/process.py", line 231, in  
_bootstrap
     self.run()
   File "/opt/csw/lib/python/multiprocessing/process.py", line 88, in  
run
     self._target(*self._args, **self._kwargs)
   File "controller.py", line 104, in work
     return subprocess.call(cmd, shell=False)
   File "/opt/csw/lib/python/subprocess.py", line 444, in call
     return Popen(*popenargs, **kwargs).wait()
   File "/opt/csw/lib/python/subprocess.py", line 595, in __init__
     errread, errwrite)
   File "/opt/csw/lib/python/subprocess.py", line 1092, in  
_execute_child
     raise child_exception
OSError: [Errno 2] No such file or directory
Process Process-2:
Traceback (most recent call last):
   File "/opt/csw/lib/python/multiprocessing/process.py", line 231, in  
_bootstrap
     self.run()
   File "/opt/csw/lib/python/multiprocessing/process.py", line 88, in  
run
     self._target(*self._args, **self._kwargs)
   File "controller.py", line 104, in work
     return subprocess.call(cmd, shell=False)
   File "/opt/csw/lib/python/subprocess.py", line 444, in call
     return Popen(*popenargs, **kwargs).wait()
   File "/opt/csw/lib/python/subprocess.py", line 595, in __init__
     errread, errwrite)
   File "/opt/csw/lib/python/subprocess.py", line 1092, in  
_execute_child
     raise child_exception
OSError: [Errno 2] No such file or directory

The files are there:

mhost{me}11% ls -la /path/to/files/BNLO_info.pf
-rw-rw-r--   1 me       group     391 May 19 22:40 /path/to/files/ 
BNLO_info.pf
myhost{me}12% ls -la /path/to/file/B11A_info.pf
-rw-rw-r--   1 me       group     391 May 19 22:27 /path/to/files/ 
B11A_info.pf

I might be doing this completely wrong, but I thought this would be  
the way to list the files dynamically. Admittedly this is just a  
stepping stone to running the actual shell script I want to run. Can  
anyone point me in the right direction or offer any advice for using  
these packages?

Thanks in advance for any help or insight.
- Rob