[Chicago] multiprocessing map input object creation

[Oren Livne]

Dear All,

I have 20,000 small IbdProblem objects that I'd like to pass to a 
process pool. Even though I'm passing a generator expression to 
pool.map, it seems that it hangs for a long time before starting to send 
them to processes. Maybe all IbdProblem objects are created before 
processes start working on them. Is that how multiprocessing behaves, or 
maybe I am missing why it hangs? Is it possible to avoid this and only 
create each IbdProblem object before it's sent to a process?

Thanks,
Oren

         manager = Manager()
         lock = manager.Lock()
         po = Pool(processes=options.num_processes)
         res = po.map(process_pair, ((IbdProblem(problem, hap1, hap2, 
None, params), options, lock)
                                     for hap1, hap2 in 
read_pairs(input_file)))

-- 
A person is just about as big as the things that make him angry.