[Numpy-discussion] Simple way to launch python processes?

[Lou Pecora]

From: Jean-Baptiste Marquette <marquett at iap.fr>

To: Discussion of Numerical Python <numpy-discussion at scipy.org> 
Sent: Wednesday, December 7, 2011 4:23 PM
Subject: Re: [Numpy-discussion] Simple way to launch python processes?
 

You should consider the powerful multiprocessing package. Have a look on this piece of code:

importglob
importos
import multiprocessing as multi
import subprocess as sub
importtime

NPROC = 4
Python = '/Library/Frameworks/EPD64.framework/Versions/Current/bin/python'
Xterm = '/usr/X11/bin/xterm '

coord = []
Size = '100x10'
XPos = 810
YPos = 170
XOffset = 0
YOffset = 0

for i in range(NPROC):
    if i % 2 == 0:
        coord.append(Size + '+' + str(YPos) + '+' + str(YOffset))
    else:
        coord.append(Size + '+' + str(XPos) + '+' + str(YOffset))
        YOffset = YOffset + YPos

def CompareColourRef(Champ):
    BaseChamp = os.path.basename(Champ)
    NameProc = int(multi.current_process().name[-1]) - 1
    print 'Processing', BaseChamp, 'on processor', NameProc+1
    os.putenv('ADAM_USER', DirWrk + 'adam_' + str(NameProc+1))
    Command =  Xterm + '-geometry ' + '"' + coord[NameProc] + '" -T " Proc' + str(NameProc+1) + ' ' + BaseChamp + ' ' + '" -e " ' + Python + ' ' + DirSrc + \
        'CompareColourRef.py ' + BaseChamp + ' 2>&1 | tee' + DirLog + BaseChamp + '.log"'
    Process = sub.Popen([Command], shell=True)
    Process.wait()
    print BaseChamp, 'processed on processor', NameProc+1
    return

pool = multi.Pool(processes=NPROC)

Champs = glob.glob(DirImg + '*/*')
results = pool.map_async(CompareColourRef, Champs)
pool.close()

while results._number_left > 0:
    print"Waiting for", results._number_left, 'tasks to complete'
    time.sleep(15)
    

pool.join()

print'Process completed'
exit(0)

Cheers
Jean-Baptiste

----------------------------------------------------------------------------------------------------------------------------------

Wow.  I will have to digest that, but thank you.

 
-- Lou Pecora, my views are my own.


________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20111207/0fd51fe9/attachment.html>