How to queue functions
Dhananjay
dhananjay.c.joshi at gmail.com
Tue Sep 18 00:56:02 EDT 2012
Dear all,
I am trying to use multiprocessing module.
I have 5 functions and 2000 input files.
First, I want to make sure that these 5 functions execute one after the
other.
Is there any way that I could queue these 5 functions within the same
script ?
Next, as there are 2000 input files.
I could queue them by queue.put() and get back to run one by one using
queue.get() as follows:
for file in files:
if '.dat.gz' in file:
q.put(file)
while True:
item = q.get()
x1 = f1(item)
x2 = f2(x1)
x3 = f3(x2)
x4 = f4(x3)
final_output = f5(x4)
However, how can I input them on my 8 core machine, so that at a time 8
files will be processed (to the set of 5 functions; each function one after
the other) ?
I am bit confused with the multiprocessing, please suggest me some
directions ....
Thank you
-- Joshi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20120918/f971ed8c/attachment.html>
More information about the Python-list
mailing list