On Tue, 23 Aug 2011 20:43:25 +0200 Charles-François Natali wrote:

...

Please consider this invitation to stick your head into an interesting problem: http://bugs.python.org/issue6721

Just for the record, I'm now in favor of the atfork mechanism. It won't solve the problem for I/O locks, but it'll at least make room for a clean and cross-library way to setup atfork handlers. I just skimmed over it, but it seemed Gregory's atfork module could be a good starting point.

Well, I would consider the I/O locks the most glaring problem. Right now, your program can freeze if you happen to do a fork() while e.g. the stderr lock is taken by another thread (which is quite common when debugging). Regards Antoine.

Le mardi 23 août 2011 à 22:07 +0200, Charles-François Natali a écrit :

2011/8/23 Antoine Pitrou :

...

Well, I would consider the I/O locks the most glaring problem. Right now, your program can freeze if you happen to do a fork() while e.g. the stderr lock is taken by another thread (which is quite common when debugging).

Indeed. To solve this, a similar mechanism could be used: after fork(), in the child process: - just reset each I/O lock (destroy/re-create the lock) if we can guarantee that the file object is in a consistent state (i.e. that all the invariants hold). That's the approach I used in my initial patch.

For I/O locks I think that would work. There could also be a process-wide "fork lock" to serialize locks and other operations, if we want 100% guaranteed consistency of I/O objects across forks.

- call a fileobject method which resets the I/O lock and sets the file object to a consistent state (in other word, an atfork handler)

I fear that the complication with atfork handlers is that you have to manage their lifecycle as well (i.e., when an IO object is destroyed, you have to unregister the handler). Regards Antoine.

On Fri, Aug 26, 2011 at 3:18 AM, Nir Aides wrote:

Another face of the discussion is about whether to deprecate the mixing of the threading and processing modules and what to do about the multiprocessing module which is implemented with worker threads.

There's a bug open - http://bugs.python.org/issue8713 which would offer non windows users the ability to avoid using fork() entirely, which would sidestep the problem outlined in the atfork() bug. Under windows, which has no fork() mechanism, we create a subprocess and then use pipes for intercommunication: nothing is inherited from the parent process except the state passed into the child. I think that "deprecating" the use of threads w/ multiprocessing - or at least crippling it is the wrong answer. Multiprocessing needs the helper threads it uses internally to manage queues, etc. Removing that ability would require a near-total rewrite, which is just a non-starter. I'd rather examine bug 8713 more closely, and offer this option for all users in 3.x and document the existing issues outlined in http://bugs.python.org/issue6721 for 2.x - the proposals in that bug are IMHO, out of bounds for a 2.x release. In essence; the issue here is multiprocessing's use of fork on unix without the following exec - which is what the windows implementation essentially does using subprocess. Adding the option to *not* fork changes the fundamental behavior on unix systems - but I fundamentally feel that it's a saner, and more consistent behavior for the module as a whole. So, I'd ask that we not talk about tearing out the ability to use MP and threads, or threads with MP - that would be crippling, and there's existing code in the wild (including multiprocessing itself) that uses this mix without issue - it's stripping out functionality for what is a surprising and painful edge case that rarely directly affects users. I would focus on the atfork() patch more directly, ignoring multiprocessing in the discussion, and focusing on the merits of gps' initial proposal and patch. jesse

On 26 Aug 2011, at 16:53, Antoine Pitrou wrote:

Hi,

...

I think that "deprecating" the use of threads w/ multiprocessing - or at least crippling it is the wrong answer. Multiprocessing needs the helper threads it uses internally to manage queues, etc. Removing that ability would require a near-total rewrite, which is just a non-starter.

I agree that this wouldn't actually benefit anyone. Besides, I don't think it's even possible to avoid threads in multiprocessing, given the various constraints. We would have to force the user to run their main thread in an event loop, and that would be twisted (tm).

...

I would focus on the atfork() patch more directly, ignoring multiprocessing in the discussion, and focusing on the merits of gps' initial proposal and patch.

I think this could also be combined with Charles-François' patch.

Regards

Have to agree with Jesse and Antoine here. Celery (celeryproject.org) uses multiprocessing, is wildly used in production, and is regarded as stable software that have been known to run for months at a time only to be restarted for software upgrades. I have been investigating an issue for some time, that I'm pretty sure is caused by this. It occurs only rarely, so rarely I have not had any actual bug reports about it, it's just something I have experienced during extensive testing. The tone of the discussion on the bug tracker makes me think that I have been very lucky :-) Using the fork+exec approach seems like a much more realistic solution than rewriting multiprocessing.Pool and Manager to not use threads. In fact this is something I have been considering as a fix for the suspected issue for for some time. It does have implications that are annoying for sure, but we are already used to this on the Windows platform (it could help portability even). -- Ask Solem twitter.com/asksol | +44 (0)7713357179

+3 (agreed to Jesse, Antoine and Ask here).  The http://bugs.python.org/issue8713 described "non-fork" implementation that always uses subprocesses rather than plain forked processes is the right way forward for multiprocessing.

I see two drawbacks: - it will be slower, since the interpreter startup time is non-negligible (well, normally you shouldn't spawn a new process for every item, but it should be noted) - it'll consume more memory, since we lose the COW advantage (even though it's already limited by the fact that even treating a variable read-only can trigger an incref, as was noted in a previous thread) cf

On Mon, 29 Aug 2011 13:03:53 -0400 Jesse Noller wrote:

2011/8/29 Charles-François Natali :

...

...

+3 (agreed to Jesse, Antoine and Ask here).  The http://bugs.python.org/issue8713 described "non-fork" implementation that always uses subprocesses rather than plain forked processes is the right way forward for multiprocessing.

I see two drawbacks: - it will be slower, since the interpreter startup time is non-negligible (well, normally you shouldn't spawn a new process for every item, but it should be noted)

Yes; but spawning and forking are both slow to begin with - it's documented (I hope heavily enough) that you should spawn multiprocessing children early, and keep them around instead of constantly creating/destroying them.

I think fork() is quite fast on modern systems (e.g. Linux). exec() is certainly slow, though. The third drawback is that you are limited to picklable objects when specifying the arguments for your child process. This can be annoying if, for example, you wanted to pass an OS resource. Regards Antoine.

Le lundi 29 août 2011 à 13:23 -0400, Jesse Noller a écrit :

Yes, it is annoying; but again - this makes it more consistent with the windows implementation. I'd rather that restriction than the "sanitization" of the ability to use threading and multiprocessing alongside one another.

That sanitization is generally useful, though. For example if you want to use any I/O after a fork(). Regards Antoine.

On Mon, Aug 29, 2011 at 8:42 PM, Jesse Noller wrote:

On Mon, Aug 29, 2011 at 1:22 PM, Antoine Pitrou wrote:

...

That sanitization is generally useful, though. For example if you want to use any I/O after a fork().

Oh! I don't disagree; I'm just against the removal of the ability to mix multiprocessing and threads; which it does internally and others do in every day code.

I am not familiar with the python-dev definition for deprecation, but when I used the word in the bug discussion I meant to advertize to users that they should not mix threading and forking since that mix is and will remain broken by design; I did not mean removal or crippling of functionality. “When I use a word,” Humpty Dumpty said, in rather a scornful tone, “it means just what I choose it to mean—neither more nor less.” - Through the Looking-Glass (btw, my tone is not scornful) And there is no way around it - the mix in general is broken, with an atfork mechanism or without it. People can choose to keep doing it in their every day code at their own risk, be it significantly high or insignificantly low. But the documentation should explain the problem clearly. As for the internal use of threads in the multiprocessing module I proposed a potential way to "sanitize" those particular worker threads: http://bugs.python.org/issue6721#msg140402 If it makes sense and entails changes to internal multiprocessing worker threads, those changes could be applied as bug fixes to Python 2.x and previous Python 3.x releases. This does not contradict adding now the feature to spawn, and to make it the only possibility in the future. I agree that this is the "saner" approach but it is a new feature not a bug fix. Nir

On Mon, Aug 29, 2011 at 8:16 PM, Antoine Pitrou wrote:

On Mon, 29 Aug 2011 13:03:53 -0400 Jesse Noller wrote:

...

Yes; but spawning and forking are both slow to begin with - it's documented (I hope heavily enough) that you should spawn multiprocessing children early, and keep them around instead of constantly creating/destroying them.

I think fork() is quite fast on modern systems (e.g. Linux). exec() is certainly slow, though.

On my system, the time it takes worker code to start is: 40 usec with thread.start_new_thread 240 usec with threading.Thread().start 450 usec with os.fork 1 ms with multiprocessing.Process.start 25 ms with subprocess.Popen to start a trivial script. so os.fork has similar latency to threading.Thread().start, while spawning is 100 times slower.