Moving away from using accelerate framework on mac os x ?

Hi, During last PyCon, Olivier Grisel (from scikits-learn fame) and myself looked into a nasty bug on mac os x: https://gist.github.com/2027412. The short story is that I believe this means numpy cannot be used with multiprocessing if linked against accelerate framework, and as such we should think about giving up on accelerate, and use e.g. ATLAS on mac for our official binaries. Long story: we recently received a answer where the engineers mention that using blas on each 'side' of a fork is not supported. The meat of the email is attached below thoughts ? David ---------- Forwarded message ---------- From: <devbugs@apple.com> Date: 2012/8/2 Subject: Bug ID 11036478: Segfault when calling dgemm with Accelerate / GCD after in a forked process To: olivier.grisel@gmail.com Hi Olivier, Thank you for contacting us regarding Bug ID# 11036478. Thank you for filing this bug report. This usage of fork() is not supported on our platform. For API outside of POSIX, including GCD and technologies like Accelerate, we do not support usage on both sides of a fork(). For this reason among others, use of fork() without exec is discouraged in general in processes that use layers above POSIX. We recommend that you either restrict usage of blas to the parent or the child process but not both, or that you switch to using GCD or pthreads rather than forking to create parallelism.

Hi David, Apple's response here is somewhat confusing, but I will add that on the supercomputing side of things we rarely fork, as this is not well-supported from the vendors or the hardware (it's hard enough to performantly spawn 500,000 processes statically, doing this dynamically becomes even more challenging). This sounds like an issue in Python multiprocessing itself, as I guess many other Apple libraries will fail or crash with the fork-no-exec model. My suggestion would be that numpy continue to integrate with Accelerate but prefer a macports or brew supplied blas, if available. This should probably also be filed as a wont-fix bug on the tracker so anybody who hits the same problem knows that it's on the system side and not us. A On Sat, Aug 4, 2012 at 2:04 PM, David Cournapeau <cournape@gmail.com> wrote:
Hi,
During last PyCon, Olivier Grisel (from scikits-learn fame) and myself looked into a nasty bug on mac os x: https://gist.github.com/2027412. The short story is that I believe this means numpy cannot be used with multiprocessing if linked against accelerate framework, and as such we should think about giving up on accelerate, and use e.g. ATLAS on mac for our official binaries.
Long story: we recently received a answer where the engineers mention that using blas on each 'side' of a fork is not supported. The meat of the email is attached below
thoughts ?
David
---------- Forwarded message ---------- From: <devbugs@apple.com> Date: 2012/8/2 Subject: Bug ID 11036478: Segfault when calling dgemm with Accelerate / GCD after in a forked process To: olivier.grisel@gmail.com
Hi Olivier,
Thank you for contacting us regarding Bug ID# 11036478.
Thank you for filing this bug report.
This usage of fork() is not supported on our platform.
For API outside of POSIX, including GCD and technologies like Accelerate, we do not support usage on both sides of a fork(). For this reason among others, use of fork() without exec is discouraged in general in processes that use layers above POSIX.
We recommend that you either restrict usage of blas to the parent or the child process but not both, or that you switch to using GCD or pthreads rather than forking to create parallelism. _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

On 08/04/2012 01:14 PM, Aron Ahmadia wrote:
Hi David,
Apple's response here is somewhat confusing, but I will add that on the supercomputing side of things we rarely fork, as this is not well-supported from the vendors or the hardware (it's hard enough to performantly spawn 500,000 processes statically, doing this dynamically becomes even more challenging). This sounds like an issue in Python multiprocessing itself, as I guess many other Apple libraries will fail or crash with the fork-no-exec model.
OpenMP is pretty widespread in supercomputing, and so is OpenMP + multiple threads using LAPACK at the same time. This does NOT sound like any issue with multiprocessing to me. Dag
My suggestion would be that numpy continue to integrate with Accelerate but prefer a macports or brew supplied blas, if available. This should probably also be filed as a wont-fix bug on the tracker so anybody who hits the same problem knows that it's on the system side and not us.
A
On Sat, Aug 4, 2012 at 2:04 PM, David Cournapeau <cournape@gmail.com <mailto:cournape@gmail.com>> wrote:
Hi,
During last PyCon, Olivier Grisel (from scikits-learn fame) and myself looked into a nasty bug on mac os x: https://gist.github.com/2027412. The short story is that I believe this means numpy cannot be used with multiprocessing if linked against accelerate framework, and as such we should think about giving up on accelerate, and use e.g. ATLAS on mac for our official binaries.
Long story: we recently received a answer where the engineers mention that using blas on each 'side' of a fork is not supported. The meat of the email is attached below
thoughts ?
David
---------- Forwarded message ---------- From: <devbugs@apple.com <mailto:devbugs@apple.com>> Date: 2012/8/2 Subject: Bug ID 11036478: Segfault when calling dgemm with Accelerate / GCD after in a forked process To: olivier.grisel@gmail.com <mailto:olivier.grisel@gmail.com>
Hi Olivier,
Thank you for contacting us regarding Bug ID# 11036478.
Thank you for filing this bug report.
This usage of fork() is not supported on our platform.
For API outside of POSIX, including GCD and technologies like Accelerate, we do not support usage on both sides of a fork(). For this reason among others, use of fork() without exec is discouraged in general in processes that use layers above POSIX.
We recommend that you either restrict usage of blas to the parent or the child process but not both, or that you switch to using GCD or pthreads rather than forking to create parallelism. _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org <mailto:NumPy-Discussion@scipy.org> http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Responding to both Dag and David,
OpenMP is pretty widespread in supercomputing, and so is OpenMP + multiple threads using LAPACK at the same time. This does NOT sound like any issue with multiprocessing to me.
These are built on dynamic thread-level (pthreads), not process-level (multiprocessing) parallelism. My original statement stands. | To be clear, I am not suggesting removing support for linking against | accelerate, just to go away from it for our binary releases. Oops, I missed that last part of your first email. If this is a problem for people, I agree that the binaries should avoid Accelerate. A

On Sat, Aug 4, 2012 at 12:14 PM, Aron Ahmadia <aron@ahmadia.net> wrote:
Hi David,
Apple's response here is somewhat confusing, but I will add that on the supercomputing side of things we rarely fork, as this is not well-supported from the vendors or the hardware (it's hard enough to performantly spawn 500,000 processes statically, doing this dynamically becomes even more challenging). This sounds like an issue in Python multiprocessing itself, as I guess many other Apple libraries will fail or crash with the fork-no-exec model.
My suggestion would be that numpy continue to integrate with Accelerate but prefer a macports or brew supplied blas, if available. This should probably also be filed as a wont-fix bug on the tracker so anybody who hits the same problem knows that it's on the system side and not us.
To be clear, I am not suggesting removing support for linking against accelerate, just to go away from it for our binary releases. David

On Sat, Aug 4, 2012 at 2:36 PM, David Cournapeau <cournape@gmail.com> wrote:
Hi David,
Apple's response here is somewhat confusing, but I will add that on the supercomputing side of things we rarely fork, as this is not well-supported from the vendors or the hardware (it's hard enough to performantly spawn 500,000 processes statically, doing this dynamically becomes even more challenging). This sounds like an issue in Python multiprocessing itself, as I guess many other Apple libraries will fail or crash with the fork-no-exec model.
My suggestion would be that numpy continue to integrate with Accelerate but prefer a macports or brew supplied blas, if available. This should
On Sat, Aug 4, 2012 at 12:14 PM, Aron Ahmadia <aron@ahmadia.net> wrote: probably
also be filed as a wont-fix bug on the tracker so anybody who hits the same problem knows that it's on the system side and not us.
To be clear, I am not suggesting removing support for linking against accelerate, just to go away from it for our binary releases.
Would there be any issues when mixing for example a numpy built against ATLAS with a scipy built against Accelerate? Ralf
participants (4)
-
Aron Ahmadia
-
Dag Sverre Seljebotn
-
David Cournapeau
-
Ralf Gommers