Using multiprocessing

nhwarriors edward.reed at gmail.com
Fri Oct 10 16:32:41 EDT 2008


I am attempting to use the (new in 2.6) multiprocessing package to
process 2 items in a large queue of items simultaneously. I'd like to
be able to print to the screen the results of each item before
starting the next one. I'm having trouble with this so far.

Here is some (useless) example code that shows how far I've gotten by
reading the documentation:

from multiprocessing import Process, Queue, current_process

def main():
	facs = []
	for i in range(50000,50005):
		facs.append(i)

	tasks = [(fac, (i,)) for i in facs]
	task_queue = Queue()
	done_queue = Queue()

	for task in tasks:
		task_queue.put(task)

	for i in range(2):
		Process(target = worker, args = (task_queue, done_queue)).start()

	for i in range(len(tasks)):
		print done_queue.get()

	for i in range(2):
		task_queue.put('STOP')

def worker(input, output):
	for func, args in iter(input.get, 'STOP'):
		result = func(*args)
		output.put(result)

def fac(n):
	f = n
	for i in range(n-1,1,-1):
		f *= i
	return 'fac('+str(n)+') done on '+current_process().name

if __name__ == '__main__':
	main()

This works great, except that nothing can be output until everything
in the queue is finished. I'd like to write out the result of fac(n)
for each item in the queue as it happens.

I'm probably approaching the problem all wrong - can anyone set me on
the right track?



More information about the Python-list mailing list