[issue19993] Pool.imap doesn't work as advertised
Tim Peters
report at bugs.python.org
Mon Dec 16 18:25:37 CET 2013
Tim Peters added the comment:
Nice to see you, Jurjen! Been a long time :-)
I'd like to see changes here too. It's unclear what "a lazy version" is intended to mean, exactly, but I agree the actual behavior is surprising, and that mpool.py is a lot less surprising in several ways.
I got bitten by this just last week, when running a parallelized search over a massive space _expected_ to succeed after exploring a tiny fraction of the search space. Ran out of system resources because imap_unordered() tried to queue up countless millions of work descriptions. I had hoped/expected that it would interleave generating and queue'ing "a few" inputs with retrieving outputs, much as mpool.py behaves.
In that case I switched to using apply_async() instead, interposing my own bounded queue (a collections.deque used only in the main program) to throttle the main program. I'm still surprised it was necessary ;-)
----------
nosy: +tim.peters
_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue19993>
_______________________________________
More information about the Python-bugs-list
mailing list