multiprocessing Pool.imap broken?

[eryksun ()]

For comparison, here's the debug info on Win32 Python 2.71. Multiprocessing on Windows seems like a very different beast (e.g. there's no way to fork).

Case 1:

In [1]: %cpaste
Pasting code; enter '--' alone on the line to stop.
:import multiprocessing as mp
:import multiprocessing.util as util
:util.log_to_stderr(util.SUBDEBUG)
:
:print(list(mp.Pool(1).imap(abs, range(3))))
:--
[DEBUG/MainProcess] created semlock with handle 604
[DEBUG/MainProcess] created semlock with handle 644
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] doing set_length()
[DEBUG/PoolWorker-1] recreated blocker with handle 36
[DEBUG/PoolWorker-1] recreated blocker with handle 48
[INFO/PoolWorker-1] child process calling self.run()
[0, 1, 2]

Case 2:

In [1]: %cpaste
Pasting code; enter '--' alone on the line to stop.
:import multiprocessing as mp
:import multiprocessing.util as util
:util.log_to_stderr(util.SUBDEBUG)
:pool=mp.Pool(1)
:print list(pool.imap(abs, range(3)))
:--
[DEBUG/MainProcess] created semlock with handle 604
[DEBUG/MainProcess] created semlock with handle 644
[DEBUG/MainProcess] added worker
[DEBUG/MainProcess] doing set_length()
[DEBUG/PoolWorker-1] recreated blocker with handle 24
[DEBUG/PoolWorker-1] recreated blocker with handle 32
[INFO/PoolWorker-1] child process calling self.run()
[0, 1, 2]


On Thursday, March 31, 2011 2:46:25 PM UTC-4, Yang Zhang wrote:
> The problem was that Pool shuts down from its finalizer:
> 
> http://stackoverflow.com/questions/5481104/multiprocessing-pool-imap-broken/5481610#5481610
> 
> On Wed, Mar 30, 2011 at 5:59 AM, eryksun () <ery... at gmail.com> wrote:
> > On Tuesday, March 29, 2011 9:44:21 PM UTC-4, Yang Zhang wrote:
> >> I've tried both the multiprocessing included in the python2.6 Ubuntu
> >> package (__version__ says 0.70a1) and the latest from PyPI (2.6.2.1).
> >> In both cases I don't know how to use imap correctly - it causes the
> >> entire interpreter to stop responding to ctrl-C's.  Any hints?  Thanks
> >> in advance.
> >>
> >> $ python
> >> Python 2.6.5 (r265:79063, Apr 16 2010, 13:57:41)
> >> [GCC 4.4.3] on linux2
> >> Type "help", "copyright", "credits" or "license" for more information.
> >> >>> import multiprocessing as mp
> >> >>> mp.Pool(1).map(abs, range(3))
> >> [0, 1, 2]
> >> >>> list(mp.Pool(1).imap(abs, range(3)))
> >> ^C^C^C^C^\Quit
> >
> > It works fine for me on Win32 Python 2.7.1 with multiprocessing 0.70a1. So it's probably an issue with the implementation on Linux.
> >
> > Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46)
> > [MSC v.1500 32 bit (Intel)] on win32
> > Type "help", "copyright", "credits" or "license" for more information.
> >>>> import multiprocessing as mp
> >>>> list(mp.Pool(1).imap(abs, range(3)))
> > [0, 1, 2]
> >>>> mp.__version__
> > '0.70a1'
> > --
> > http://mail.python.org/mailman/listinfo/python-list
> >
> 
> 
> 
> -- 
> Yang Zhang
> http://yz.mit.edu/