[New-bugs-announce] [issue13751] multiprocessing.pool hangs if any worker raises an Exception whose constructor requires a parameter

[Faheem Mitha]

New submission from Faheem Mitha <faheem at faheem.info>:

See my question at http://stackoverflow.com/questions/8785899/hang-in-python-script-using-sqlalchemy-and-multiprocessing

I can't improve on the analysis by Lorenzo Bolla,
so I reproduce his example below. This example hangs if
BadExc is thrown, but not if GoodExc is thrown.
The only difference between these is that the GoodExc does
not require an argument be passed to its constructor, while
BadExc does.

This looks like a bug to me, though I suppose it might
be some pickling limitation.

I have confirmed this behavior is present in 2.6.6, 2.7.2,
and 3.1.3, all tested on Debian squeeze.

                                       Regards, Faheem

#################################################

import multiprocessing

class BadExc(Exception):
    def __init__(self, a):
        '''Non-optional param in the constructor.'''
        self.a = a

class GoodExc(Exception):
    def __init__(self, a=None):
        '''Optional param in the constructor.'''
        self.a = a

def do(kwargs):
    i = kwargs['i']
    print i
    raise BadExc('a')
    # raise GoodExc('a')
    return i

pool = multiprocessing.Pool(processes=5)
results = []
arglist = []
for i in range(10):
    arglist.append({'i':i})
r = pool.map_async(do, arglist, callback=results.append)
try:
    # set a timeout in order to be able to catch C-c
    r.get(1e100)
except KeyboardInterrupt:
    pass
print results

----------
components: Library (Lib)
messages: 150975
nosy: fmitha
priority: normal
severity: normal
status: open
title: multiprocessing.pool hangs if any worker raises an Exception whose constructor requires a parameter
type: behavior
versions: Python 2.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13751>
_______________________________________