[Numpy-discussion] Dropping support for Accelerate

Nathaniel Smith njs at pobox.com
Tue Jul 25 09:19:58 EDT 2017


I updated the bit about OpenBLAS wheel with some more information on
the status of that work. It's not super important, but FYI.

I also want to disagree with this characterization of the
Accelerate/multiprocessing issue: "This problem was due to a bug in
multiprocessing and is fixed in Python 3.4 and later; Accelerate was
POSIX compliant but multiprocessing was not."

In 3.4 it became possible to *work around* this issue, but it requires
configuring the multiprocessing module in a non-default way, which
means that the common end-user experience is still that they try using
multiprocessing, and they get random hangs with no other feedback, and
then spend hours or days debugging before they discover this
configuration option. (And the problem occurs on MacOS only, so you
get extra fun when e.g. a module is developed on Windows or Linux and
then you give it to a less-technical collaborator on MacOS and it
breaks on their computer and you have no idea why.)

And the workaround is suboptimal -- fork()'s memory-sharing semantics
are very powerful. I've had cases where I could easily and efficiently
solve a problem using multiprocessing in fork() mode, but where
enabling the workaround for Accelerate would have made it impossible.
(Specifically this happened because I had a huge read-only data
structure that I could load once in the parent process, and then all
the child processes could share it through fork's virtual memory
magic; I didn't have enough memory to load two copies of it, yet fork
let me have 10 or 20 virtual copies.)

Technically, yes, mixing threads and fork can't be done in a
POSIX-compliant manner. But no-one runs their code on an abstract
POSIX machine, and on actual systems it's totally possible to make
this work reliably. OpenBLAS does it. Users don't care if Apple is
technically correct, they just want their stuff to work.

-n

On Tue, Jul 25, 2017 at 4:57 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> On Sun, Jul 23, 2017 at 5:07 PM, Ilhan Polat <ilhanpolat at gmail.com> wrote:
>> Ouch, that's from 2012 :(  I'll add this thread as a reference to the wiki
>> list.
>>
>>
>> On Sun, Jul 23, 2017 at 5:22 PM, Nathan Goldbaum <nathan12343 at gmail.com>
>> wrote:
>>>
>>> See
>>> https://mail.scipy.org/pipermail/numpy-discussion/2012-August/063589.html
>>> and replies in that thread.
>>>
>>> Quote from an Apple engineer in that thread:
>>>
>>> "For API outside of POSIX, including GCD and technologies like Accelerate,
>>> we do not support usage on both sides of a fork(). For this reason among
>>> others, use of fork() without exec is discouraged in general in processes
>>> that use layers above POSIX."
>>>
>>> On Sun, Jul 23, 2017 at 10:16 AM, Ilhan Polat <ilhanpolat at gmail.com>
>>> wrote:
>>>>
>>>> That's probably because I know nothing about the issue, is there any
>>>> reference I can read about?
>>>>
>>>> But in general, please feel free populate new items in the wiki page.
>>>>
>>>> On Sun, Jul 23, 2017 at 11:15 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>>>>
>>>>> I've been wishing we'd stop shipping Accelerate for years, because of
>>>>> how it breaks multiprocessing – that doesn't seem to be on your list
>>>>> yet.
>>>>>
>>>>> On Sat, Jul 22, 2017 at 3:50 AM, Ilhan Polat <ilhanpolat at gmail.com>
>>>>> wrote:
>>>>> > A few months ago, I had the innocent intention to wrap LDLt
>>>>> > decomposition
>>>>> > routines of LAPACK into SciPy but then I am made aware that the
>>>>> > minimum
>>>>> > required version of LAPACK/BLAS was due to Accelerate framework. Since
>>>>> > then
>>>>> > I've been following the core SciPy team and others' discussion on this
>>>>> > issue.
>>>>> >
>>>>> > We have been exchanging opinions for quite a while now within various
>>>>> > SciPy
>>>>> > issues and PRs about the ever-increasing Accelerate-related issues and
>>>>> > I've
>>>>> > compiled a brief summary about the ongoing discussions to reduce the
>>>>> > clutter.
>>>>> >
>>>>> > First, I would like to kindly invite everyone to contribute and
>>>>> > sharpen the
>>>>> > cases presented here
>>>>> >
>>>>> > https://github.com/scipy/scipy/wiki/Dropping-support-for-Accelerate
>>>>> >
>>>>> > The reason I specifically wanted to post this also in NumPy mailing
>>>>> > list is
>>>>> > to probe for the situation from the NumPy-Accelerate perspective. Is
>>>>> > there
>>>>> > any NumPy specific problem that would indirectly effect SciPy should
>>>>> > the
>>>>> > support for Accelerate is dropped?
>>>>> >
>>>>> >
>>>>> >
>>>>> >
>>>>> > _______________________________________________
>>>>> > NumPy-Discussion mailing list
>>>>> > NumPy-Discussion at python.org
>>>>> > https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Nathaniel J. Smith -- https://vorpus.org
>>>>> _______________________________________________
>>>>> NumPy-Discussion mailing list
>>>>> NumPy-Discussion at python.org
>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> NumPy-Discussion at python.org
>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>
> I added some more discussion, and some links to previous discussion on
> the mailing list.
>
> I also pointed to this PR :
> https://github.com/MacPython/numpy-wheels/pull/1 - which builds
> OpenBLAS wheels for numpy.  The same kind of thing would work fine for
> Scipy.
>
> Cheers,
>
> Matthew
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion



-- 
Nathaniel J. Smith -- https://vorpus.org


More information about the NumPy-Discussion mailing list