multiprocessing & itertools.product Iterator

Christian mining.facts at
Sun Mar 25 00:35:29 CET 2012


I struggle to "extend" a multiprocessing example to my problem with a
itertools.product result iterator.
How I have to  assign the elements approriate to
Pool.imap/calc functions?

Thanks in advance

from multiprocessing import Process,Queue,Pool
import Calculation
import DataSimulation
from itertools import product

def produce_example_combos(size=6,na=1000,nb=10):
    data = DataSimulation.DataSimulation()
    a = [data.generate_simple(size) for x in xrange(na)]
    b = [data.generate_simple(size) for x in xrange(nb)]
    it = product(a,b)
    return it

def calc(elements):
    calc.q.put("Doing:" +  elements[0] + elements[1])
    ratio = Calculation.ratio(elements[0],elements[1])
    return ratio

def calc_init(q):
    calc.q = q

if __name__ == '__main__':
    combos = produce_example_combos()
    print "tesdata generated"
    q = Queue()
    p = Pool(10, calc_init, [q])
    results = p.imap(calc,

    for i in combos:
        print q.get()

More information about the Python-list mailing list