[Python-Dev] Usage of the multiprocessing API and object lifetime

[Antoine Pitrou]

On Tue, 11 Dec 2018 17:22:49 +0100
Victor Stinner <vstinner at redhat.com> wrote:
> Le mar. 11 déc. 2018 à 17:06, Antoine Pitrou <solipsis at pitrou.net> a écrit :
> > > We are not talking about simple strings, but processes and threads.  
> >
> > Right, but do those have an impact on the program's correctness, or
> > simply on its performance (or memory consumption)?  
> 
> Performance.

Well, at least we shouldn't emit ResourceWarning for performance issues.
So if someone used "with pool:", they shouldn't get a ResourceWarning
afterwards, even if some threads are still not finished running.

> I don't know if these changes are similar to my questions about
> multiprocessing API, since socketserver didn't expose the list of
> processes/threads and doesn't provide a method to wait until they
> complete. Well... ForkingMixIn has an *undocumented*
> collect_children() which is non-blocking by default (I added a
> keyword-only 'blocking' parameter).

socketserver is not the same as a multiprocessing Pool *at all*.  The
usage will be vastly different between the two.  Just because we did
this change for socketserver is not enough of a reason to do it for
multiprocessing too.

> Another example: the close() method of an asyncio event loop doesn't
> wait until threads/processes complete:
> 
> "asyncio: BaseEventLoop.close() shutdowns the executor without waiting
> causing leak of dangling threads"
> https://bugs.python.org/issue34037

It's not a leak though, it's a resource that's collected a bit later.
It may annoy the test machinery (and this could be a reason to add
private or public APIs to help finish the threads), but it shouldn't be
an actual issue for user code.

Here as well, I think you should be careful not to introduce annoyances
(unwanted latencies) in user code.

Regards

Antoine.