Hi David,
Apple's response here is somewhat confusing, but I will add that on the supercomputing side of things we rarely fork, as this is not well-supported from the vendors or the hardware (it's hard enough to performantly spawn 500,000 processes statically, doing this dynamically becomes even more challenging). This sounds like an issue in Python multiprocessing itself, as I guess many other Apple libraries will fail or crash with the fork-no-exec model.
My suggestion would be that numpy continue to integrate with Accelerate but prefer a macports or brew supplied blas, if available. This should probably also be filed as a wont-fix bug on the tracker so anybody who hits the same problem knows that it's on the system side and not us.
A
On Sat, Aug 4, 2012 at 2:04 PM, David Cournapeau
<cournape@gmail.com> wrote:
Hi,
During last PyCon, Olivier Grisel (from scikits-learn fame) and myself
looked into a nasty bug on mac os x: https://gist.github.com/2027412.
The short story is that I believe this means numpy cannot be used with
multiprocessing if linked against accelerate framework, and as such we
should think about giving up on accelerate, and use e.g. ATLAS on mac
for our official binaries.
Long story: we recently received a answer where the engineers mention
that using blas on each 'side' of a fork is not supported. The meat of
the email is attached below
thoughts ?
David
---------- Forwarded message ----------
From: <devbugs@apple.com>
Date: 2012/8/2
Subject: Bug ID 11036478: Segfault when calling dgemm with Accelerate
/ GCD after in a forked process
To: olivier.grisel@gmail.com
Hi Olivier,
Thank you for contacting us regarding Bug ID# 11036478.
Thank you for filing this bug report.
This usage of fork() is not supported on our platform.
For API outside of POSIX, including GCD and technologies like
Accelerate, we do not support usage on both sides of a fork(). For
this reason among others, use of fork() without exec is discouraged in
general in processes that use layers above POSIX.
We recommend that you either restrict usage of blas to the parent or
the child process but not both, or that you switch to using GCD or
pthreads rather than forking to create parallelism.
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion