[New-bugs-announce] [issue25053] Possible race condition in Pool

[Stanislaw Izaak Pitucha]

New submission from Stanislaw Izaak Pitucha:

This is something that happened once and I cannot reproduce anymore. In an IPython session I've done the following (see session below), which resulted in pool.map raising its internal exceptions in cases where it shouldn't. Then it worked again. (see in/out 29)

Running exactly the same lines again did not result in the same behaviour. I tried multiple times in the same session as well as new sessions. Looks like a weird race / corruption.

I'm on Ubuntu's '3.4.3 (default, Mar 26 2015, 22:03:40) \n[GCC 4.9.2]'. Running with 2 virtualised cores (vmware). 64-bit system.

In [21]: import multiprocessing
In [22]: p=multiprocessing.Pool()
... (unrelated, checked p.map? and other helps)
In [26]: class A(object):
    def a(self, i):
        print("boo", i, multiprocessing.current_process())
   ....: 
In [27]: obj = A()
In [28]: p.map(obj.a, [1,2,3,4])
Process ForkPoolWorker-1:
Process ForkPoolWorker-2:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/lib/python3.4/multiprocessing/process.py", line 254, in _bootstrap
    self.run()
  File "/usr/lib/python3.4/multiprocessing/process.py", line 254, in _bootstrap
    self.run()
  File "/usr/lib/python3.4/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.4/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.4/multiprocessing/queues.py", line 357, in get
    return ForkingPickler.loads(res)
AttributeError: Can't get attribute 'A' on <module '__main__'>
  File "/usr/lib/python3.4/multiprocessing/queues.py", line 357, in get
    return ForkingPickler.loads(res)
AttributeError: Can't get attribute 'A' on <module '__main__'>
boo 3 <ForkProcess(ForkPoolWorker-3, started daemon)>
boo 4 <ForkProcess(ForkPoolWorker-3, started daemon)>

^CProcess ForkPoolWorker-4:
Process ForkPoolWorker-3:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/lib/python3.4/multiprocessing/process.py", line 254, in _bootstrap
    self.run()
  File "/usr/lib/python3.4/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.4/multiprocessing/process.py", line 254, in _bootstrap
    self.run()
  File "/usr/lib/python3.4/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.4/multiprocessing/queues.py", line 354, in get
    with self._rlock:
  File "/usr/lib/python3.4/multiprocessing/queues.py", line 355, in get
    res = self._reader.recv_bytes()
  File "/usr/lib/python3.4/multiprocessing/synchronize.py", line 96, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
  File "/usr/lib/python3.4/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/usr/lib/python3.4/multiprocessing/connection.py", line 416, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib/python3.4/multiprocessing/connection.py", line 383, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-28-3be6fbcc359b> in <module>()
----> 1 p.map(obj.a, [1,2,3,4])

/usr/lib/python3.4/multiprocessing/pool.py in map(self, func, iterable, chunksize)
    258         in a list that is returned.
    259         '''
--> 260         return self._map_async(func, iterable, mapstar, chunksize).get()
    261 
    262     def starmap(self, func, iterable, chunksize=None):

/usr/lib/python3.4/multiprocessing/pool.py in get(self, timeout)
    591 
    592     def get(self, timeout=None):
--> 593         self.wait(timeout)
    594         if not self.ready():
    595             raise TimeoutError

/usr/lib/python3.4/multiprocessing/pool.py in wait(self, timeout)
    588 
    589     def wait(self, timeout=None):
--> 590         self._event.wait(timeout)
    591 
    592     def get(self, timeout=None):

/usr/lib/python3.4/threading.py in wait(self, timeout)
    551             signaled = self._flag
    552             if not signaled:
--> 553                 signaled = self._cond.wait(timeout)
    554             return signaled
    555         finally:

/usr/lib/python3.4/threading.py in wait(self, timeout)
    288         try:    # restore state no matter what (e.g., KeyboardInterrupt)
    289             if timeout is None:
--> 290                 waiter.acquire()
    291                 gotit = True
    292             else:

KeyboardInterrupt: 

In [29]: p.map(obj.a, [1,2,3,4])
boo 1 <ForkProcess(ForkPoolWorker-5, started daemon)>
boo 2 <ForkProcess(ForkPoolWorker-6, started daemon)>
boo 3 <ForkProcess(ForkPoolWorker-5, started daemon)>
boo 4 <ForkProcess(ForkPoolWorker-6, started daemon)>
Out[29]: [None, None, None, None]

----------
components: Library (Lib)
messages: 250362
nosy: Stanislaw Izaak Pitucha
priority: normal
severity: normal
status: open
title: Possible race condition in Pool
type: behavior
versions: Python 3.4

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue25053>
_______________________________________