[Python-Dev] Usage of the multiprocessing API and object lifetime

[Antoine Pitrou]

Hi,

On Tue, 11 Dec 2018 15:21:31 +0100
Victor Stinner <vstinner at redhat.com> wrote:
> 
> Pablo's issue35378 evolved to add a weak reference in iterators to try
> to detect when the Pool is destroyed: raise an exception from the
> iterator, if possible.

That's an ok fix for me.

> By the way, I'm surprised that "with pool:" doesn't release all
> resources.

That's not a problem, as long as the destructor _does_ release
resources.

> From a technical point of view, I would prefer to become stricter.

Using "with pool:" is fine, we shouldn't start raising a warning for it.

What you are proposing here starts to smell like an anti-pattern to
me.  Python _is_ a garbage-collected language, so by definition, there
_are_ going to be resources that are automatically collected when an
object disappears.  If I'm allocating a 2GB bytes object, then PyPy may
delay the deallocation much longer than CPython.  Do you propose we add
a release() method to bytes objects to avoid this issue (and emit a
warning for people who don't call release() on bytes objects)?

You can't change the language's philosophy.  We warn about open files
because those have user-visible consequences (such as unflushed
buffers, or not being able to delete the file on Windows).  If there is
no user-visible consequence to not calling join() on a Pool, then we
shouldn't warn about it.

Regards

Antoine.