[Numpy-discussion] Windows wheels, built, but should we deploy?

josef.pktd at gmail.com josef.pktd at gmail.com
Fri Mar 4 22:30:31 EST 2016


On Fri, Mar 4, 2016 at 1:38 PM, Matthew Brett <matthew.brett at gmail.com>
wrote:

> On Fri, Mar 4, 2016 at 12:29 AM, David Cournapeau <cournape at gmail.com>
> wrote:
> >
> >
> > On Fri, Mar 4, 2016 at 4:42 AM, Matthew Brett <matthew.brett at gmail.com>
> > wrote:
> >>
> >> Hi,
> >>
> >> Summary:
> >>
> >> I propose that we upload Windows wheels to pypi.  The wheels are
> >> likely to be stable and relatively easy to maintain, but will have
> >> slower performance than other versions of numpy linked against faster
> >> BLAS / LAPACK libraries.
> >>
> >> Background:
> >>
> >> There's a long discussion going on at issue github #5479 [1], where
> >> the old problem of Windows wheels for numpy came up.
> >>
> >> For those of you not following this issue, the current situation for
> >> community-built numpy Windows binaries is dire:
> >>
> >> * We have not so far provided windows wheels on pypi, so `pip install
> >> numpy` on Windows will bring you a world of pain;
> >> * Until recently we did provide .exe "superpack" installers on
> >> sourceforge, but these became increasingly difficult to build and we
> >> gave up building them as of the latest (1.10.4) release.
> >>
> >> Despite this, popularity of Windows wheels on pypi is high.   A few
> >> weeks ago, Donald Stufft ran a query for the binary wheels most often
> >> downloaded from pypi, for any platform [2] . The top five most
> >> downloaded were (n_downloads, name):
> >>
> >> 6646,
> >>
> numpy-1.10.4-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
> >> 5445, cryptography-1.2.1-cp27-none-win_amd64.whl
> >> 5243, matplotlib-1.4.0-cp34-none-win32.whl
> >> 5241, scikit_learn-0.15.1-cp34-none-win32.whl
> >> 4573, pandas-0.17.1-cp27-none-win_amd64.whl
> >>
> >> So a) the OSX numpy wheel is very popular and b) despite the fact that
> >> we don't provide a numpy wheel for Windows, matplotlib, sckit_learn
> >> and pandas, that depend on numpy, are the 3rd, 4th and 5th most
> >> downloaded wheels as of a few weeks ago.
> >>
> >> So, there seems to be a large appetite for numpy wheels.
> >>
> >> Current proposal:
> >>
> >> I have now built numpy wheels, using the ATLAS blas / lapack library -
> >> the build is automatic and reproducible [3].
> >>
> >> I chose ATLAS to build against, rather than, say OpenBLAS, because
> >> we've had some significant worries in the past about the reliability
> >> of OpenBLAS, and I thought it better to err on the side of
> >> correctness.
> >>
> >> However, these builds are relatively slow for matrix multiply and
> >> other linear algebra routines compared numpy built against OpenBLAS or
> >> MKL (which we cannot use because of its license) [4].   In my very
> >> crude array test of a dot product and matrix inversion, the ATLAS
> >> wheels were 2-3 times slower than MKL.  Other benchmarks on Julia
> >> found about the same result for ATLAS vs OpenBLAS on 32-bit bit, but a
> >> much bigger difference on 64-bit (for an earlier version of ATLAS than
> >> we are currently using) [5].
> >>
> >> So, our numpy wheels likely to be stable and give correct results, but
> >> will be somewhat slow for linear algebra.
> >
> >
> > I would not worry too much about this: at worst, this gives us back the
> > situation where we were w/ so-called superpack, which have been
> successful
> > in the past to spread numpy use on windows.
> >
> > My main worry is whether this locks us into ATLAS  for a long time
> because
> > of package depending on numpy blas/lapack (scipy, scikit learn). I am not
> > sure how much this is the case.
>
> You mean the situation where other packages try to find the BLAS /
> LAPACK library and link against that?   My impression was that neither
> scipy or scikit-learn do that at the moment, but I'm happy to be
> corrected.
>
> You'd know better than me about this, but my understanding is that
> BLAS / LAPACK has a standard interface that should allow code to run
> the same way, regardless of which BLAS / LAPACK library it is linking
> to.  So, even if another package is trying to link against the numpy
> BLAS, swapping the numpy BLAS library shouldn't cause a problem
> (unless the package is trying to link to ATLAS-specific stuff, which
> seems a bit unlikely).
>
> Is that right?
>


AFAIK, numpy doesn't provide access to BLAS/LAPACK. scipy does. statsmodels
is linking to the installed BLAS/LAPACK in cython code through scipy. So
far we haven't seen problems with different versions. I think scipy
development works very well to isolate linalg library version specific
parts from the user interface.

AFAIU, The main problem will be linking to inconsistent Fortran libraries
in downstream packages that use Fortran.
Eg. AFAIU it won't work to pip install a ATLAS based numpy and then install
a MKL based scipy from Gohlke.

I don't know if there is a useful error message, or if this just results in
puzzled users.

Josef



>
> Cheers,
>
> Matthew
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160304/0808ed36/attachment.html>


More information about the NumPy-Discussion mailing list