An error in multiprocessing on MacOSX?
See this: http://mail.scipy.org/pipermail/numpy-discussion/2012-August/063593.html According to Apple enineers: """ For API outside of POSIX, including GCD and technologies like Accelerate, we do not support usage on both sides of a fork(). For this reason among others, use of fork() without exec is discouraged in general in processes that use layers above POSIX. """ Multiprocessing on OSX calls os.fork, but not os.exec. Thus, is multiprocessing errorneously implemented on Mac? Forking without calling exec means that only APIs inside POSIX can be used by the child process. For NumPy, it even affects functions like matrix multiplication when the accelerate framework is used for BLAS. Does multiprocessing needs a reimplementation on Mac to behave as it does on Windows? (Yes it would cripple it similarly to the crippled multiprocessing on Windows.) And what about Python itself? Is there any non-POSIX code in the interpreter? If it is, os.fork should be removed on Mac. Sturla
On Wed, 21 Nov 2012 16:12:04 +0100 Sturla Molden <sturla@molden.no> wrote:
Or perhaps "fork()" is erroneously implemented on Mac. Regardless, http://bugs.python.org/issue8713 has a proposal by Richard to make things more configurable on all POSIX platforms. Regards Antoine.
From: Antoine Pitrou <solipsis@pitrou.net> To: python-ideas@python.org
No, it's not that fork is erroneously implemented, it's that CoreFoundation, as designed, doesn't work across a properly-implemented POSIX fork. And presumably the same is true for various other Apple technologies, but they won't give a complete list of what is and isn't safe and why—instead, they just say "don't use any non-POSIX stuff if you fork without exec". And the problem Python users face isn't that multiprocessing works differently on OS X vs. FreeBSD or linux, but that their programs may be quietly using non-fork-safe things like Accelerate.framework on OS X but not on FreeBSD or linux.
On 21 Nov, 2012, at 20:25, Antoine Pitrou <solipsis@pitrou.net> wrote:
Fork works fine, its "just" that most system libraries above the POSIX layer don't bother to clean up their state in the child proces. There may be good reasons for that (cleaning up state changes by background threads might be hard), but its pretty annoying non the less. Ronald
On Wed, Nov 21, 2012 at 9:12 AM, Sturla Molden <sturla@molden.no> wrote:
That isn't the way I read the quoted statement - and it's probably not true. The way I read it, you have to make all your above-the-POSIX calls either before you fork, or make them all *after* you fork, but not on "both sides of a fork()." The reality is probably that you're ok so long as you make sure the above-the-POSIX data is in the "right" state before you fork. Rather than trying to describe the "right" state, Apple provides a simple rule to do that - and then provides a simple-minded rule to force compliance with that rule. Note that the same warning applies to some of the objects that are defined by POSIX interfaces. If you fork with them in the "wrong" state, you're going to get broken behavior.
-1. Mac's make nice Unix desktops (if you like their GUI), and it's not uncommon to find people writing & testing software on Mac's for deployment to POSIX servers. Such people using the multiprocessing module need it to provide proper POSIX behavior.
And what about Python itself? Is there any non-POSIX code in the interpreter? If it is, os.fork should be removed on Mac.
Well, the Mac-specific portions might. But those of us developing for deployment to non-Mac systems won't be using those. In general, these issues are simply a case of needing to be aware of what your program is doing, and making sure you don't do things that can cause problems - whether you're using above-the-POSIX Mac APIs, or POSIX the problematic Posix APIs. A case might be made that the multiprocessing module should help with that, by providing a way to say "I'm doing things that make fork dangerous, please use the system-appropriate fork+exec call instead." <mike
On Wed, Nov 21, 2012 at 11:34 AM, Mike Meyer <mwm@mired.org> wrote:
I don't care how this is read or what the default behavior is. All I want is for someone who cares about multiprocessing actually working well for people to implement optional support for *not* using os.fork() on posixish systems. ie: port an equivalent of the windows stuff over. I don't use multiprocessing so I've never looked into adding it. It shouldn't be difficult given the legwork has already been done for use on windows. this is exactly what issue8713 is asking for.
+10 - though I wouldn't call it a "re" implementation, just a port of the windows stuff. Mac's make nice Unix desktops (if you like their GUI), and it's not
Those people are delusional, they need to test their software on the OS it is destined to run on. Not on their terminal.
Yep. Like I said, I don't personally care what the default is. I just want support for both options so that people have a way to _not_ shoot themselves in the foot. -gps
On Wed, Nov 21, 2012 at 2:29 PM, Gregory P. Smith <greg@krypto.org> wrote:
Are you saying that it's delusional to try and write portable code that runs on POSIX-compliant platforms in Python? Of course, any such attempt always has to be tested against the platforms it's supposed to run on - after all, they may fail to implement the standard correctly. But those are generally considered to be bugs in the platform in question that need to be worked around, *not* bugs in software. Since Apple is one of the few vendors releasing products with a Unix certification, their products are prime candidates as development platforms for anyone working on Unix software. <mike
On 21/11/2012 8:29pm, Gregory P. Smith wrote:
An implementation is available at http://hg.python.org/sandbox/sbt#spawn You just need to stick multiprocessing.set_start_method('spawn') at the beginning of the program to use fork+exec instead of fork. The test suite passes. (I would not say that making this work was that straightforward though.) That branch also supports the starting of processes via a server process. That gives an alternative solution to the problem of mixing fork() with threads, but has the advantage of being as fast as the default fork start method. However, it does not work on systems where fd passing is unsupported like OpenSolaris. -- Richard
Den 21. nov. 2012 kl. 23:18 skrev Richard Oudkerk <shibturn@gmail.com>:
That is very nice, thank you :-) BTW, fd passing is possible on Windows too, using DuplicateHandle. One can "inject" an open file handle into a different process, but some means of ipc (e.g. a pipe) must be used to communicate it's value. Sturla
On 26/11/2012 9:23pm, Sturla Molden wrote:
BTW, fd passing is possible on Windows too, using DuplicateHandle. One can "inject" an open file handle into a different process, but some means of ipc (e.g. a pipe) must be used to communicate it's value.
multiprocessing on Windows already depends on that feature;-) -- Richard
On 27/11/2012 12:26am, Sturla Molden wrote:
In 3.3 on Windows, connection objects can be pickled using a trick with DuplicateHandle() and DUPLICATE_CLOSE_SOURCE. The same could be done for locks fairly easily. But on Unix I cannot see a way to reliably manage the life time of picklable locks. -- Richard
On 26.11.2012 23:11, Richard Oudkerk wrote:
multiprocessing on Windows already depends on that feature;-)
Indeed it does :) But it seems the handle duplication is only used when the Popen class is initiated, so it is not more flexible than just inheriting handles on fork or CreateProcess. It would be nice to pass newly created fds to child processes that are already running. I.e. what I would like to see is an advanced queue that can be used to pass files, sockets, locks, and other objects associated with a handle. That is, when a "special object" on the queue is deserialized (unpickled) by the receiver, it sends a request back to the sender for handle duplication. One obvious use case would be a "thread pool" design for a server app using processes instead of threads. Sturla
On 27.11.2012 18:03, Sturla Molden wrote:
Actually, it seems the non-Windows versions still use os.fork for inheriting fds. So fd passing is only used on Windows, and the use of fd passing on the Windows version is only used to simulate handle inheritance (which CreateProcess can do as well). It seems its use is held back by the use of fork to inherit fds on non-Windows versions. If the fd passing was used in a specialized queue, e.g. called multiprocessing.fdqueue, the means of startup should not matter. Sturla
On 27/11/2012 5:03pm, Sturla Molden wrote:
Socket pickling is already supported in 3.3. Adding the same support for file objects would be easy enough. But I don't see a sensible way to support general pickling of lock objects on Unix. So I don't much like the idea of adding support only for Windows. -- Richard
Den 27. nov. 2012 kl. 18:44 skrev Richard Oudkerk <shibturn@gmail.com>:
But I don't see a sensible way to support general pickling of lock objects on Unix. So I don't much like the idea of adding support only for Windows.
I would suggest to use a piece of shared memory and atomic compare-and-swap. Shared memory can be pickled (e.g. take a look at what I have on github). Sturla
On 27/11/2012 9:37pm, Sturla Molden wrote:
I would suggest to use a piece of shared memory and atomic compare-and-swap. Shared memory can be pickled (e.g. take a look at what I have on github).
On unix (without assuming a recent gcc) there isn't any cross platform way of doing compare-and-swap or other atomic operations. And even if there were, you need to worry about busy-waiting. One could also consider using a process-shared semaphore or mutex allocated from shared memory, but that does not seem to be available on many platforms. A simple (but wasteful) alternative would be to build a lock/semaphore from a pipe: write to release and read to acquire. Then you can use normal fd passing. -- Richard
On Tue, Nov 27, 2012 at 4:53 PM, Richard Oudkerk <shibturn@gmail.com> wrote:
Do we need a cross-platform solution for all posix systems, or just a wrapper that provides that functionality that can be implemented on many of them? Something akin to the apr global mutex routines? <mike
On 21 Nov, 2012, at 16:12, Sturla Molden <sturla@molden.no> wrote:
And what about Python itself? Is there any non-POSIX code in the interpreter? If it is, os.fork should be removed on Mac.
Not necessarily in the interpeter itself, but the proxy-detection code in _scproxy uses non-POSIX code for detecting the user's proxy settings. BTW. removing os.fork is overkill, some system APIs don't work properly on OSX after fork and complain loudly when you try to use them. So don't do that. Ronald
On Wed, Nov 21, 2012 at 12:49 PM, Ronald Oussoren <ronaldoussoren@mac.com>wrote:
well, it depends. its not right to ask for "non-posix code" as the restrictions of what you can use after a fork() are related to what you've done before the fork (as someone else stated). if your process has spawned threads, the entire python interpreter is unsafe to use after a fork() as posix requires that a limited subset of safe functions are all that is used from then on...
On Wed, Nov 21, 2012 at 2:57 PM, Gregory P. Smith <greg@krypto.org> wrote:
If your process has spawned threads, POSIX fork() is unsafe. It creates a clone of your process with every thread but the one calling fork() stopped dead. Unless you can guarantee that this was a safe thing to do then (i.e. - all your other threads hold no locks, have no shared data structures in violation of invariant, etc.), you can be hosed. Some cases can even hose the parent process. Calling exec shortly after doing the fork will help with some of these issues. Not all of them. <mike
On Wed, 21 Nov 2012 16:12:04 +0100 Sturla Molden <sturla@molden.no> wrote:
Or perhaps "fork()" is erroneously implemented on Mac. Regardless, http://bugs.python.org/issue8713 has a proposal by Richard to make things more configurable on all POSIX platforms. Regards Antoine.
From: Antoine Pitrou <solipsis@pitrou.net> To: python-ideas@python.org
No, it's not that fork is erroneously implemented, it's that CoreFoundation, as designed, doesn't work across a properly-implemented POSIX fork. And presumably the same is true for various other Apple technologies, but they won't give a complete list of what is and isn't safe and why—instead, they just say "don't use any non-POSIX stuff if you fork without exec". And the problem Python users face isn't that multiprocessing works differently on OS X vs. FreeBSD or linux, but that their programs may be quietly using non-fork-safe things like Accelerate.framework on OS X but not on FreeBSD or linux.
On 21 Nov, 2012, at 20:25, Antoine Pitrou <solipsis@pitrou.net> wrote:
Fork works fine, its "just" that most system libraries above the POSIX layer don't bother to clean up their state in the child proces. There may be good reasons for that (cleaning up state changes by background threads might be hard), but its pretty annoying non the less. Ronald
On Wed, Nov 21, 2012 at 9:12 AM, Sturla Molden <sturla@molden.no> wrote:
That isn't the way I read the quoted statement - and it's probably not true. The way I read it, you have to make all your above-the-POSIX calls either before you fork, or make them all *after* you fork, but not on "both sides of a fork()." The reality is probably that you're ok so long as you make sure the above-the-POSIX data is in the "right" state before you fork. Rather than trying to describe the "right" state, Apple provides a simple rule to do that - and then provides a simple-minded rule to force compliance with that rule. Note that the same warning applies to some of the objects that are defined by POSIX interfaces. If you fork with them in the "wrong" state, you're going to get broken behavior.
-1. Mac's make nice Unix desktops (if you like their GUI), and it's not uncommon to find people writing & testing software on Mac's for deployment to POSIX servers. Such people using the multiprocessing module need it to provide proper POSIX behavior.
And what about Python itself? Is there any non-POSIX code in the interpreter? If it is, os.fork should be removed on Mac.
Well, the Mac-specific portions might. But those of us developing for deployment to non-Mac systems won't be using those. In general, these issues are simply a case of needing to be aware of what your program is doing, and making sure you don't do things that can cause problems - whether you're using above-the-POSIX Mac APIs, or POSIX the problematic Posix APIs. A case might be made that the multiprocessing module should help with that, by providing a way to say "I'm doing things that make fork dangerous, please use the system-appropriate fork+exec call instead." <mike
On Wed, Nov 21, 2012 at 11:34 AM, Mike Meyer <mwm@mired.org> wrote:
I don't care how this is read or what the default behavior is. All I want is for someone who cares about multiprocessing actually working well for people to implement optional support for *not* using os.fork() on posixish systems. ie: port an equivalent of the windows stuff over. I don't use multiprocessing so I've never looked into adding it. It shouldn't be difficult given the legwork has already been done for use on windows. this is exactly what issue8713 is asking for.
+10 - though I wouldn't call it a "re" implementation, just a port of the windows stuff. Mac's make nice Unix desktops (if you like their GUI), and it's not
Those people are delusional, they need to test their software on the OS it is destined to run on. Not on their terminal.
Yep. Like I said, I don't personally care what the default is. I just want support for both options so that people have a way to _not_ shoot themselves in the foot. -gps
On Wed, Nov 21, 2012 at 2:29 PM, Gregory P. Smith <greg@krypto.org> wrote:
Are you saying that it's delusional to try and write portable code that runs on POSIX-compliant platforms in Python? Of course, any such attempt always has to be tested against the platforms it's supposed to run on - after all, they may fail to implement the standard correctly. But those are generally considered to be bugs in the platform in question that need to be worked around, *not* bugs in software. Since Apple is one of the few vendors releasing products with a Unix certification, their products are prime candidates as development platforms for anyone working on Unix software. <mike
On 21/11/2012 8:29pm, Gregory P. Smith wrote:
An implementation is available at http://hg.python.org/sandbox/sbt#spawn You just need to stick multiprocessing.set_start_method('spawn') at the beginning of the program to use fork+exec instead of fork. The test suite passes. (I would not say that making this work was that straightforward though.) That branch also supports the starting of processes via a server process. That gives an alternative solution to the problem of mixing fork() with threads, but has the advantage of being as fast as the default fork start method. However, it does not work on systems where fd passing is unsupported like OpenSolaris. -- Richard
Den 21. nov. 2012 kl. 23:18 skrev Richard Oudkerk <shibturn@gmail.com>:
That is very nice, thank you :-) BTW, fd passing is possible on Windows too, using DuplicateHandle. One can "inject" an open file handle into a different process, but some means of ipc (e.g. a pipe) must be used to communicate it's value. Sturla
On 26/11/2012 9:23pm, Sturla Molden wrote:
BTW, fd passing is possible on Windows too, using DuplicateHandle. One can "inject" an open file handle into a different process, but some means of ipc (e.g. a pipe) must be used to communicate it's value.
multiprocessing on Windows already depends on that feature;-) -- Richard
On 27/11/2012 12:26am, Sturla Molden wrote:
In 3.3 on Windows, connection objects can be pickled using a trick with DuplicateHandle() and DUPLICATE_CLOSE_SOURCE. The same could be done for locks fairly easily. But on Unix I cannot see a way to reliably manage the life time of picklable locks. -- Richard
On 26.11.2012 23:11, Richard Oudkerk wrote:
multiprocessing on Windows already depends on that feature;-)
Indeed it does :) But it seems the handle duplication is only used when the Popen class is initiated, so it is not more flexible than just inheriting handles on fork or CreateProcess. It would be nice to pass newly created fds to child processes that are already running. I.e. what I would like to see is an advanced queue that can be used to pass files, sockets, locks, and other objects associated with a handle. That is, when a "special object" on the queue is deserialized (unpickled) by the receiver, it sends a request back to the sender for handle duplication. One obvious use case would be a "thread pool" design for a server app using processes instead of threads. Sturla
On 27.11.2012 18:03, Sturla Molden wrote:
Actually, it seems the non-Windows versions still use os.fork for inheriting fds. So fd passing is only used on Windows, and the use of fd passing on the Windows version is only used to simulate handle inheritance (which CreateProcess can do as well). It seems its use is held back by the use of fork to inherit fds on non-Windows versions. If the fd passing was used in a specialized queue, e.g. called multiprocessing.fdqueue, the means of startup should not matter. Sturla
On 27/11/2012 5:03pm, Sturla Molden wrote:
Socket pickling is already supported in 3.3. Adding the same support for file objects would be easy enough. But I don't see a sensible way to support general pickling of lock objects on Unix. So I don't much like the idea of adding support only for Windows. -- Richard
Den 27. nov. 2012 kl. 18:44 skrev Richard Oudkerk <shibturn@gmail.com>:
But I don't see a sensible way to support general pickling of lock objects on Unix. So I don't much like the idea of adding support only for Windows.
I would suggest to use a piece of shared memory and atomic compare-and-swap. Shared memory can be pickled (e.g. take a look at what I have on github). Sturla
On 27/11/2012 9:37pm, Sturla Molden wrote:
I would suggest to use a piece of shared memory and atomic compare-and-swap. Shared memory can be pickled (e.g. take a look at what I have on github).
On unix (without assuming a recent gcc) there isn't any cross platform way of doing compare-and-swap or other atomic operations. And even if there were, you need to worry about busy-waiting. One could also consider using a process-shared semaphore or mutex allocated from shared memory, but that does not seem to be available on many platforms. A simple (but wasteful) alternative would be to build a lock/semaphore from a pipe: write to release and read to acquire. Then you can use normal fd passing. -- Richard
On Tue, Nov 27, 2012 at 4:53 PM, Richard Oudkerk <shibturn@gmail.com> wrote:
Do we need a cross-platform solution for all posix systems, or just a wrapper that provides that functionality that can be implemented on many of them? Something akin to the apr global mutex routines? <mike
On 21 Nov, 2012, at 16:12, Sturla Molden <sturla@molden.no> wrote:
And what about Python itself? Is there any non-POSIX code in the interpreter? If it is, os.fork should be removed on Mac.
Not necessarily in the interpeter itself, but the proxy-detection code in _scproxy uses non-POSIX code for detecting the user's proxy settings. BTW. removing os.fork is overkill, some system APIs don't work properly on OSX after fork and complain loudly when you try to use them. So don't do that. Ronald
On Wed, Nov 21, 2012 at 12:49 PM, Ronald Oussoren <ronaldoussoren@mac.com>wrote:
well, it depends. its not right to ask for "non-posix code" as the restrictions of what you can use after a fork() are related to what you've done before the fork (as someone else stated). if your process has spawned threads, the entire python interpreter is unsafe to use after a fork() as posix requires that a limited subset of safe functions are all that is used from then on...
On Wed, Nov 21, 2012 at 2:57 PM, Gregory P. Smith <greg@krypto.org> wrote:
If your process has spawned threads, POSIX fork() is unsafe. It creates a clone of your process with every thread but the one calling fork() stopped dead. Unless you can guarantee that this was a safe thing to do then (i.e. - all your other threads hold no locks, have no shared data structures in violation of invariant, etc.), you can be hosed. Some cases can even hose the parent process. Calling exec shortly after doing the fork will help with some of these issues. Not all of them. <mike
participants (7)
-
Andrew Barnert -
Antoine Pitrou -
Gregory P. Smith -
Mike Meyer -
Richard Oudkerk -
Ronald Oussoren -
Sturla Molden