Mailman 3 OS X wheels: speed versus multiprocessing - NumPy-Discussion

7 Apr 2015


      Hi all,
Starting with 1.9.1, the official numpy OS X wheels (the ones you get by
doing "pip install numpy") have been built to use Apple's Accelerate
library for linear algebra. This is fast, but it breaks multiprocessing in
obscure ways (e.g. see this user report:
https://github.com/numpy/numpy/issues/5752).
Unfortunately, there is no obvious best solution to what linear algebra
package to use, so we have to make a decision as to which set of
compromises we prefer.
Options:
Accelerate: fast, but breaks multiprocessing as above.
OpenBLAS: fast, but Julian raised concerns about its trustworthiness last
year (
http://mail.scipy.org/pipermail/numpy-discussion/2014-March/069659.html).
Possibly things have improved since then (I get the impression that they've
gotten some additional developer attention from the Julia community), but I
don't know.
Atlas: slower (faster than reference blas but definitely slower than fancy
options like the above), but solid.
My feeling is that for wheels in particular it's more important that
everything "just work" than that we get the absolute fastest speeds. And
this is especially true for the multiprocessing issue, given that it's a
widely used part of the stdlib, the failures are really obscure/confusing,
and there is no workaround for python 2 which is still where a majority of
our users still are. So I'd vote for using either atlas or OpenBLAS. (And
would defer to Julian and Matthew about which to choose between these.)
Any opinions, objections?
-n