I updated the bit about OpenBLAS wheel with some more information on the status of that work. It's not super important, but FYI. I also want to disagree with this characterization of the Accelerate/multiprocessing issue: "This problem was due to a bug in multiprocessing and is fixed in Python 3.4 and later; Accelerate was POSIX compliant but multiprocessing was not." In 3.4 it became possible to *work around* this issue, but it requires configuring the multiprocessing module in a non-default way, which means that the common end-user experience is still that they try using multiprocessing, and they get random hangs with no other feedback, and then spend hours or days debugging before they discover this configuration option. (And the problem occurs on MacOS only, so you get extra fun when e.g. a module is developed on Windows or Linux and then you give it to a less-technical collaborator on MacOS and it breaks on their computer and you have no idea why.) And the workaround is suboptimal -- fork()'s memory-sharing semantics are very powerful. I've had cases where I could easily and efficiently solve a problem using multiprocessing in fork() mode, but where enabling the workaround for Accelerate would have made it impossible. (Specifically this happened because I had a huge read-only data structure that I could load once in the parent process, and then all the child processes could share it through fork's virtual memory magic; I didn't have enough memory to load two copies of it, yet fork let me have 10 or 20 virtual copies.) Technically, yes, mixing threads and fork can't be done in a POSIX-compliant manner. But no-one runs their code on an abstract POSIX machine, and on actual systems it's totally possible to make this work reliably. OpenBLAS does it. Users don't care if Apple is technically correct, they just want their stuff to work. -n On Tue, Jul 25, 2017 at 4:57 AM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Sun, Jul 23, 2017 at 5:07 PM, Ilhan Polat <ilhanpolat@gmail.com> wrote:
Ouch, that's from 2012 :( I'll add this thread as a reference to the wiki list.
On Sun, Jul 23, 2017 at 5:22 PM, Nathan Goldbaum <nathan12343@gmail.com> wrote:
See https://mail.scipy.org/pipermail/numpy-discussion/2012-August/063589.html and replies in that thread.
Quote from an Apple engineer in that thread:
"For API outside of POSIX, including GCD and technologies like Accelerate, we do not support usage on both sides of a fork(). For this reason among others, use of fork() without exec is discouraged in general in processes that use layers above POSIX."
On Sun, Jul 23, 2017 at 10:16 AM, Ilhan Polat <ilhanpolat@gmail.com> wrote:
That's probably because I know nothing about the issue, is there any reference I can read about?
But in general, please feel free populate new items in the wiki page.
On Sun, Jul 23, 2017 at 11:15 AM, Nathaniel Smith <njs@pobox.com> wrote:
I've been wishing we'd stop shipping Accelerate for years, because of how it breaks multiprocessing – that doesn't seem to be on your list yet.
On Sat, Jul 22, 2017 at 3:50 AM, Ilhan Polat <ilhanpolat@gmail.com> wrote:
A few months ago, I had the innocent intention to wrap LDLt decomposition routines of LAPACK into SciPy but then I am made aware that the minimum required version of LAPACK/BLAS was due to Accelerate framework. Since then I've been following the core SciPy team and others' discussion on this issue.
We have been exchanging opinions for quite a while now within various SciPy issues and PRs about the ever-increasing Accelerate-related issues and I've compiled a brief summary about the ongoing discussions to reduce the clutter.
First, I would like to kindly invite everyone to contribute and sharpen the cases presented here
https://github.com/scipy/scipy/wiki/Dropping-support-for-Accelerate
The reason I specifically wanted to post this also in NumPy mailing list is to probe for the situation from the NumPy-Accelerate perspective. Is there any NumPy specific problem that would indirectly effect SciPy should the support for Accelerate is dropped?
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
-- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
I added some more discussion, and some links to previous discussion on the mailing list.
I also pointed to this PR : https://github.com/MacPython/numpy-wheels/pull/1 - which builds OpenBLAS wheels for numpy. The same kind of thing would work fine for Scipy.
Cheers,
Matthew _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
-- Nathaniel J. Smith -- https://vorpus.org