[Numpy-discussion] Simple way to launch python processes?

[Jean-Baptiste Marquette]

You should consider the powerful multiprocessing package. Have a look on this piece of code:

import glob
import os
import multiprocessing as multi
import subprocess as sub
import time

NPROC = 4
Python = '/Library/Frameworks/EPD64.framework/Versions/Current/bin/python'
Xterm = '/usr/X11/bin/xterm '

coord = []
Size = '100x10'
XPos = 810
YPos = 170
XOffset = 0
YOffset = 0

for i in range(NPROC):
    if i % 2 == 0:
        coord.append(Size + '+' + str(YPos) + '+' + str(YOffset))
    else:
        coord.append(Size + '+' + str(XPos) + '+' + str(YOffset))
        YOffset = YOffset + YPos

def CompareColourRef(Champ):
    BaseChamp = os.path.basename(Champ)
    NameProc = int(multi.current_process().name[-1]) - 1
    print 'Processing', BaseChamp, 'on processor', NameProc+1
    os.putenv('ADAM_USER', DirWrk + 'adam_' + str(NameProc+1))
    Command =  Xterm + '-geometry ' + '"' + coord[NameProc] + '" -T " Proc' + str(NameProc+1) + ' ' + BaseChamp + ' ' + '" -e " ' + Python + ' ' + DirSrc + \
        'CompareColourRef.py ' + BaseChamp + ' 2>&1 | tee ' + DirLog + BaseChamp + '.log"'
    Process = sub.Popen([Command], shell=True)
    Process.wait()
    print BaseChamp, 'processed on processor', NameProc+1
    return

pool = multi.Pool(processes=NPROC)

Champs = glob.glob(DirImg + '*/*')
results = pool.map_async(CompareColourRef, Champs)
pool.close()

while results._number_left > 0:
    print "Waiting for", results._number_left, 'tasks to complete'
    time.sleep(15)
    
pool.join()

print 'Process completed'
exit(0)

Cheers
Jean-Baptiste


Le 7 déc. 2011 à 15:43, Olivier Delalleau a écrit :

> Maybe try stackoverflow, since this isn't really a numpy question.
> To run a command like "python myscript.py arg1 arg2" in a separate process, you can do:
>     p = subprocess.Popen("python myscript.py arg1 arg2".split())
> You can launch many of these, and if you want to know if a process p is over, you can call p.poll().
> I'm sure there are other (and better) options though.
> 
> -=- Olivier
> 
> 2011/12/7 Lou Pecora <lou_boog2000 at yahoo.com>
> I would like to launch python modules or functions (I don't know which is easier to do, modules or functions) in separate Terminal windows so I can see the output from each as they execute.  I need to be able to pass each module or function a set of parameters.  I would like to do this from a python script already running in a Terminal window.  In other words, I'd start up a "master" script and it would launch, say, three processes using another module or a function with different parameter values for each launch and each would run independently in its own Terminal window so stdout from each process would go to it's own respective window.  When the process terminated the window would remain open.
> 
> I've begun to look at subprocess modules, etc., but that's pretty confusing. I can do what I say above manually, but it's gotten clumsy as I want to run eventually in 12 cores.
> 
> I have a Mac Pro running Mac OS X 10.6.
> 
> If there is a better forum to ask this question, please let me know. 
> 
> Thanks for any advice.
> 
>  
> -- Lou Pecora, my views are my own.
> 
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> 
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20111207/329867c7/attachment.html>