[Python-ideas] An error in multiprocessing on MacOSX?

[Mike Meyer]

On Wed, Nov 21, 2012 at 9:12 AM, Sturla Molden <sturla at molden.no> wrote:
> See this:
>
> http://mail.scipy.org/pipermail/numpy-discussion/2012-August/063593.html
>
> According to Apple engineers:
>
> """
> For API outside of POSIX, including GCD and technologies like
> Accelerate, we do not support usage on both sides of a fork(). For
> this reason among others, use of fork() without exec is discouraged in
> general in processes that use layers above POSIX.
> """
>
> Multiprocessing on OSX calls os.fork, but not os.exec.
>
> Thus, is multiprocessing errorneously implemented on Mac? Forking without
> calling exec means that only APIs inside POSIX can be used by the child
> process.

That isn't the way I read the quoted statement - and it's probably not
true. The way I read it, you have to make all your above-the-POSIX
calls either before you fork, or make them all *after* you fork, but
not on "both sides of a fork()." The reality is probably that you're
ok so long as you make sure the above-the-POSIX data is in the "right"
state before you fork. Rather than trying to describe the "right"
state, Apple provides a simple rule to do that - and then provides a
simple-minded rule to force compliance with that rule.

Note that the same warning applies to some of the objects that are
defined by POSIX interfaces. If you fork with them in the "wrong"
state, you're going to get broken behavior.

> For NumPy, it even affects functions like matrix multiplication when the
> accelerate framework is used for BLAS.
>
> Does multiprocessing needs a reimplementation on Mac to behave as it does on
> Windows? (Yes it would cripple it similarly to the crippled multiprocessing
> on Windows.)

-1.

Mac's make nice Unix desktops (if you like their GUI), and it's not
uncommon to find people writing & testing software on Mac's for
deployment to POSIX servers. Such people using the multiprocessing
module need it to provide proper POSIX behavior.

> And what about Python itself? Is there any non-POSIX code in the
> interpreter? If it is, os.fork should be removed on Mac.

Well, the Mac-specific portions might. But those of us developing for
deployment to non-Mac systems won't be using those.

In general, these issues are simply a case of needing to be aware of
what your program is doing, and making sure you don't do things that
can cause problems - whether you're using above-the-POSIX Mac APIs, or
POSIX the problematic Posix APIs.  A case might be made that the
multiprocessing module should help with that, by providing a way to
say "I'm doing things that make fork dangerous, please use the
system-appropriate fork+exec call instead."

       <mike