[New-bugs-announce] [issue21201] Uninformative error message in multiprocessing.Manager()

[Wojciech Walczak]

New submission from Wojciech Walczak:

While using multiprocessing.Manager() to send data between processes I've noticed that one of the traceback messages is not very informative:

Traceback (most recent call last):
  File "age_predict.py", line 39, in <module>
    train_data, train_target, test_data = prepare_data()
  File "age_predict.py", line 30, in prepare_data
    train_data = q.get(True, 10000)
  File "<string>", line 2, in get
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 777, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------
Unserializable message: ('#RETURN', [A WHOLE LOT OF DATA GOES HERE]
 

The real problem here is that my machine is running out of memory, but the traceback message shadows this information and reports "Unserializable message" instead.

After a simple path (see: attached file) the traceback message is as follows:


Traceback (most recent call last):
  File "age_predict.py", line 39, in <module>
    train_data, train_target, test_data = prepare_data()
  File "age_predict.py", line 30, in prepare_data
    train_data = q.get(True, 10000)
  File "<string>", line 2, in get
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 775, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------
Unserializable message: Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 288, in serve_client
    send(msg)
MemoryError: out of memory


Here's another example (see: http://bugs.python.org/issue6766):

This python3 code:

from multiprocessing import Manager
manager = Manager()
ns_proxy = manager.Namespace()
evt_proxy = manager.Event()
ns_proxy.my_event_proxy = evt_proxy
print(ns_proxy.my_event_proxy)


Produces following traceback message:


Traceback (most recent call last):
  File "test.py", line 6, in <module>
    print(ns_proxy.my_event_proxy)
  File "/cpython/Lib/multiprocessing/managers.py", line 1032, in __getattr__
    return callmethod('__getattribute__', (key,))
  File "/cpython/Lib/multiprocessing/managers.py", line 748, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
----------------------------------------------------------------------
Unserializable message: ('#RETURN', <threading.Event object at 0x7f9484aff278>)
----------------------------------------------------------------------


...and after applying the attached patch it becomes clearer what's going on:


Traceback (most recent call last):
  File "test.py", line 6, in <module>
    print(ns_proxy.my_event_proxy)
  File "/cpython/Lib/multiprocessing/managers.py", line 1032, in __getattr__
    return callmethod('__getattribute__', (key,))
  File "/cpython/Lib/multiprocessing/managers.py", line 748, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
----------------------------------------------------------------------
Unserializable message: Traceback (most recent call last):
  File "/cpython/Lib/multiprocessing/managers.py", line 276, in serve_client
    send(msg)
  File "/cpython/Lib/multiprocessing/connection.py", line 206, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "/cpython/Lib/multiprocessing/reduction.py", line 50, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class '_thread.lock'>: attribute lookup lock on _thread failed

This patch doesn't break any tests.

----------
components: Library (Lib)
files: managers_tb_msg.patch
keywords: patch
messages: 215934
nosy: wojtekwalczak
priority: normal
severity: normal
status: open
title: Uninformative error message in multiprocessing.Manager()
type: behavior
versions: Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5
Added file: http://bugs.python.org/file34785/managers_tb_msg.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue21201>
_______________________________________