[issue5331] multiprocessing hangs when Pool used within Process

[Gabriel de Perthuis]

Gabriel de Perthuis <onyxg7 at users.sourceforge.net> added the comment:

Apparently the pool workers die all at once, just after the pool creates
them and before the pool is used.
I added a few lines to multiprocessing/pool.py to get the stack and the
exception backtrace.

        except (EOFError, IOError):
            import traceback
            debug(traceback.format_exc())
            debug(''.join(traceback.format_stack()))
            debug('worker got EOFError or IOError -- exiting')
            break


INFO::Rule dispatcher::multiprocessing::child process calling self.run()
DEBUG::Rule dispatcher::multiprocessing::created semlock with handle
3082559488
DEBUG::Rule dispatcher::multiprocessing::created semlock with handle
3082104832
DEBUG::Rule dispatcher::multiprocessing::created semlock with handle
3081826304
DEBUG::Rule dispatcher::multiprocessing::created semlock with handle
3081822208
INFO::PoolWorker-3:1::multiprocessing::child process calling self.run()
DEBUG::PoolWorker-3:1::multiprocessing::Traceback (most recent call last):
  File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py",
line 57, in worker
    task = get()
  File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/queues.py",
line 339, in get
    return recv()
IOError: [Errno 9] Bad file descriptor

DEBUG::PoolWorker-3:1::multiprocessing::  File
"/home/who/var/co/git-svn/what-base/correlator/bin/correlator", line 30,
in <module>
    what.corr.actors.main.main()
  File
"/home/who/var/co/git-svn/what-base/correlator/lib/what/corr/actors/main.py",
line 47, in main
    rrp.start()
  File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 109, in start
    self._popen = Popen(self)
  File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/forking.py",
line 99, in __init__
    code = process_obj._bootstrap()
  File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 93, in run
    self._target(*self._args, **self._kwargs)
  File
"/home/who/var/co/git-svn/what-base/correlator/lib/what/corr/actors/rule_dispatcher.py",
line 26, in main
    initargs=(manager.agg_msgs_queue, ))
  File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/__init__.py",
line 232, in Pool
    return Pool(processes, initializer, initargs)
  File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py",
line 107, in __init__
    w.start()
  File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 109, in start
    self._popen = Popen(self)
  File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/forking.py",
line 99, in __init__
    code = process_obj._bootstrap()
  File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 93, in run
    self._target(*self._args, **self._kwargs)
  File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py",
line 61, in worker
    debug(''.join(traceback.format_stack()))

DEBUG::PoolWorker-3:1::multiprocessing::worker got EOFError or IOError
-- exiting
INFO::PoolWorker-3:1::multiprocessing::process shutting down
DEBUG::PoolWorker-3:1::multiprocessing::running all "atexit" finalizers
with priority >= 0

----------
nosy: +onyxg7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5331>
_______________________________________