[New-bugs-announce] [issue9592] Limitations in objects returned by multiprocessing Pool

[Freek Dijkstra]

New submission from Freek Dijkstra <freek at macfreek.nl>:

I came across three limitation in the multiprocessing module that were not handled correctly.

Attached is a file that reproduces the errors in minimal code. I tested them with Python 2.6.5 and 3.1.2.

Expected result:
multiprocessing.Pool's promises a map function where each result is returned transparently to the main process (despite that the calculation was done in a subprocess)

Actual result:
Not all values returned by a subprocess are returned transparently.
I expected multiprocessing to handle these cases gracefully by yielding an Exception in the Main process.

The cases I found are:

1) A multiprocessing worker can not return (return, not raise!) an Exception. 
If this is attempted, the result handler thread in the Pool calls the exception with no arguments, 
which might raise an error if multiple arguments are required:
TypeError: ('__init__() takes exactly 2 arguments (1 given)', <class '__main__.MyException'>, ())    


2) A multiprocessing worker can not return an hashlib Object.
If this is attempted, pickle returns an error:
PicklingError: Can't pickle <type '_hashlib.HASH'>: attribute lookup _hashlib.HASH failed


3) A multiprocessing worker can not return an Object which overrides __getattr__, and accesses a variable from self in __getattr__.
If this is attempted, Python 2.6 crashes with a bus error:
Program terminated by uncaught signal #10 after 1.56 seconds.
Python 3.1 yields the error:
RuntimeError: maximum recursion depth exceeded while calling a Python object

----------
components: Library (Lib)
files: multiprocessingbugs.py
messages: 113816
nosy: macfreek
priority: normal
severity: normal
status: open
title: Limitations in objects returned by multiprocessing Pool
type: crash
versions: Python 2.6, Python 3.1
Added file: http://bugs.python.org/file18507/multiprocessingbugs.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue9592>
_______________________________________