[Numpy-discussion] Moving away from using accelerate framework on mac os x ?

[Dag Sverre Seljebotn]

On 08/04/2012 01:14 PM, Aron Ahmadia wrote:
> Hi David,
>
> Apple's response here is somewhat confusing, but I will add that on the
> supercomputing side of things we rarely fork, as this is not
> well-supported from the vendors or the hardware (it's hard enough to
> performantly spawn 500,000 processes statically, doing this dynamically
> becomes even more challenging).  This sounds like an issue in Python
> multiprocessing itself, as I guess many other Apple libraries will fail
> or crash with the fork-no-exec model.

OpenMP is pretty widespread in supercomputing, and so is OpenMP + 
multiple threads using LAPACK at the same time. This does NOT sound like 
any issue with multiprocessing to me.

Dag


>
> My suggestion would be that numpy continue to integrate with Accelerate
> but prefer a macports or brew supplied blas, if available.  This should
> probably also be filed as a wont-fix bug on the tracker so anybody who
> hits the same problem knows that it's on the system side and not us.
>
> A
>
> On Sat, Aug 4, 2012 at 2:04 PM, David Cournapeau <cournape at gmail.com
> <mailto:cournape at gmail.com>> wrote:
>
>     Hi,
>
>     During last PyCon, Olivier Grisel (from scikits-learn fame) and myself
>     looked into a nasty bug on mac os x: https://gist.github.com/2027412.
>     The short story is that I believe this means numpy cannot be used with
>     multiprocessing if linked against accelerate framework, and as such we
>     should think about giving up on accelerate, and use e.g. ATLAS on mac
>     for our official binaries.
>
>     Long story: we recently received a answer where the engineers mention
>     that using blas on each 'side' of a fork is not supported. The meat of
>     the email is attached below
>
>     thoughts ?
>
>     David
>
>     ---------- Forwarded message ----------
>     From:  <devbugs at apple.com <mailto:devbugs at apple.com>>
>     Date: 2012/8/2
>     Subject: Bug ID 11036478: Segfault when calling dgemm with Accelerate
>     / GCD after in a forked process
>     To: olivier.grisel at gmail.com <mailto:olivier.grisel at gmail.com>
>
>
>     Hi Olivier,
>
>     Thank you for contacting us regarding Bug ID# 11036478.
>
>     Thank you for filing this bug report.
>
>     This usage of fork() is not supported on our platform.
>
>     For API outside of POSIX, including GCD and technologies like
>     Accelerate, we do not support usage on both sides of a fork(). For
>     this reason among others, use of fork() without exec is discouraged in
>     general in processes that use layers above POSIX.
>
>     We recommend that you either restrict usage of blas to the parent or
>     the child process but not both, or that you switch to using GCD or
>     pthreads rather than forking to create parallelism.
>     _______________________________________________
>     NumPy-Discussion mailing list
>     NumPy-Discussion at scipy.org <mailto:NumPy-Discussion at scipy.org>
>     http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>