[Python-Dev] Usage of the multiprocessing API and object lifetime

Pablo Galindo Salgado pablogsal at gmail.com
Tue Dec 11 14:18:22 EST 2018


>
> Your original solution would have added a strong reference back to the
> pool from the iterator. At first glance, that seems like a reasonable
> solution to me. Victor is worried about the "risk of new reference
> cycles". But reference cycles are not a problem - we have the cyclic
> GC precisely to deal with them. So I'd like to see a better
> justification for rejecting that solution than "there might be
> reference cycles". But in response to that, you made the iterator have
> a weak reference back to the pool. That's flawed because it doesn't
> prevent the pool being terminated - as you say, the deadlock is still
> present.


Just to be clear: I am in favour of the strong reference, but I also
understand the "danger" (leaking the pool until the pool's generation
reaches the threshold and the gc runs) and that is the reason I was
experimenting with the weakreference.

On Tue, 11 Dec 2018 at 18:37, Paul Moore <p.f.moore at gmail.com> wrote:

> On Tue, 11 Dec 2018 at 17:50, Pablo Galindo Salgado <pablogsal at gmail.com>
> wrote:
> > I agree that misusage of the pool should not be encouraged but in this
> situation the fact that
> > this code hangs:
> >
> > import multiprocessing
> >
> > for x in multiprocessing.Pool().imap(int, ["4", "3"]):
> >     print(x)
> >
> >
> > is a bit worriying because although is incorrect and an abuse of the
> API, users can do this easily with
> > no error message other than a misterious hang.
>
> OK, so the first problem here (to me, at least) is that it's not
> obvious *why* this code is incorrect and an abuse of the API. It takes
> a reasonable amount of thinking about the problem to notice that the
> Pool object isn't retained, but the iterator returned from imap is.
> And when the pool is collected, the worker processes are terminated,
> causing the hang, as the worker never sends a result back to the main
> process. But it's not obvious to me why the pool is collected before
> the imap method has completed.
>
> As I understand it, originally the code worked because the pool
> *didn't* call terminate() when collected. Now it does, and we have a
> problem. I'm not *entirely* sure why, if the pool is terminated, the
> wait in the iterator doesn't terminate immediately with some sort of
> "process being waited on died" error, but let's assume there are good
> reasons for that (as I mentioned before, I'm not an expert in
> multiprocessing, so I'm OK with assuming that the original design,
> done by people who *are* experts, is sound :-))
>
> Your original solution would have added a strong reference back to the
> pool from the iterator. At first glance, that seems like a reasonable
> solution to me. Victor is worried about the "risk of new reference
> cycles". But reference cycles are not a problem - we have the cyclic
> GC precisely to deal with them. So I'd like to see a better
> justification for rejecting that solution than "there might be
> reference cycles". But in response to that, you made the iterator have
> a weak reference back to the pool. That's flawed because it doesn't
> prevent the pool being terminated - as you say, the deadlock is still
> present.
>
> > I have found this on several places and people were
> > very confused because usually the interpreter throws some kind of error
> indication. In my humble opinion,
> > we should try to avoid hanging as a consequence of the misusage,
> whatever we do.
>
> I agree with this. But that implies to me that we should be holding a
> strong reference to the pool,
>
> As a (somewhat weak) analogy, consider
>
>     for n in map(int, ["1", "2"]):
>         print(n)
>
> That won't fail if the list gets collected, because map keeps a
> reference to the list. My intuition would be that the Pool().imap
> example would hold a reference to the pool on essentially the same
> basis.
>
> The more I think about this, the more I struggle to see Victor's logic
> for rejecting your original solution. And I *certainly* don't see why
> this issue should justify changing the whole API to require users to
> explicitly manage pool lifetimes.
>
> Paul
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20181211/a6a34f65/attachment.html>


More information about the Python-Dev mailing list