[Python-Dev] Usage of the multiprocessing API and object lifetime

[Pablo Galindo Salgado]

> > Pablo's issue35378 evolved to add a weak reference in iterators to try
> > to detect when the Pool is destroyed: raise an exception from the
> > iterator, if possible.

> That's an ok fix for me.

I am playing with weakreferences inside the iterator and result objects,
but this may not be enough/a complete solution. For example, take the
code of ApplyResult.get:

def get(self, timeout=None):
    if self._pool() is None:   # self._pool is a weakref
        raise RuntimeError("The pool is dead. Aborting") <--- new code
    self.wait(timeout)

It can be that the pool is alive when we check for it (self._pool() is
None) but while
the code is waiting with no timeout (timeout=None) the pool dies,
effectively leaving the
program deadlocked with no error.

--

I agree that misusage of the pool should not be encouraged but in this
situation the fact that
this code hangs:

import multiprocessing

for x in multiprocessing.Pool().imap(int, ["4", "3"]):
    print(x)


is a bit worriying because although is incorrect and an abuse of the
API, users can do this easily with
no error message other than a misterious hang. I have found this on
several places and people were
very confused because usually the interpreter throws some kind of
error indication. In my humble opinion,
we should try to avoid hanging as a consequence of the misusage, whatever we do.

Pablo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181211/5a6ee3b2/attachment.html>