[New-bugs-announce] [issue17836] multiprocessing exceptions with useful traceback

[Andres Riancho]

New submission from Andres Riancho:

In pool.py, the worker function reads as follows:

http://svn.python.org/view/python/trunk/Lib/multiprocessing/pool.py?view=markup
"""
68	        job, i, func, args, kwds = task
69	        try:
70	            result = (True, func(*args, **kwds))
71	        except Exception, e:
72	            result = (False, e)
...
488	        if self._success:
489	            return self._value
490	        else:
491	            raise self._value
"""

If an exception is raised in the function you sent to the pool, the exception you get has "raise self._value" as the last line; which is correct but useless for debugging.

mp_exception_bug.py reproduces this error. This is the output:

"""
Traceback (most recent call last):
  File "mp_exception_bug.py", line 8, in <module>
    print p.map(f, [1,2,3])
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 227, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 528, in get
    raise self._value
NameError: global name 'y' is not defined
"""

As you can imagine, "NameError: global name 'y' is not defined" is not enough in complex projects.

If we apply some changes to the pool.py we could get something similar to this:

"""
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 98, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 67, in mapstar
    return map(*args)
  File "mp_exception_bug.py", line 4, in f
    return x*y
NameError: global name 'y' is not defined

Traceback (most recent call last):
  File "mp_exception_bug.py", line 8, in <module>
    print p.map(f, [1,2,3])
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 231, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 535, in get
    raise self._value[0]
NameError: global name 'y' is not defined
"""

The patch is simple but ugly:

"""
> import sys
> import traceback
72c100,102
<             result = (False, e)
---
>             exc_info = sys.exc_info()
>             tb_string = traceback.format_exc(exc_info[2])
>             result = (False, (e, tb_string))
491c532,535
<             raise self._value
---
>             # Do something with the exception here, the simplest (but ugliest)
>             # thing to do is to simply print it to the console
>             print self._value[1]
>             raise self._value[0]
"""

Note that I tried to replace the "raise self._value[0]" with a raise with three parameters, being the last one the traceback we get using "exc_info = sys.exc_info()" but sadly it is not possible to pickle tracebacks.

I understand that printing is not the best thing to do here, but I wanted to get this discussion started and come to a real solution.

Thanks

----------
components: Library (Lib)
files: mp_exception_bug.py
messages: 187751
nosy: Andres.Riancho
priority: normal
severity: normal
status: open
title: multiprocessing exceptions with useful traceback
versions: Python 3.5
Added file: http://bugs.python.org/file30009/mp_exception_bug.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue17836>
_______________________________________