[Numpy-discussion] Windows wheels, built, but should we deploy?

Matthew Brett matthew.brett at gmail.com
Fri Mar 4 13:38:22 EST 2016


On Fri, Mar 4, 2016 at 12:29 AM, David Cournapeau <cournape at gmail.com> wrote:
>
>
> On Fri, Mar 4, 2016 at 4:42 AM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
>>
>> Hi,
>>
>> Summary:
>>
>> I propose that we upload Windows wheels to pypi.  The wheels are
>> likely to be stable and relatively easy to maintain, but will have
>> slower performance than other versions of numpy linked against faster
>> BLAS / LAPACK libraries.
>>
>> Background:
>>
>> There's a long discussion going on at issue github #5479 [1], where
>> the old problem of Windows wheels for numpy came up.
>>
>> For those of you not following this issue, the current situation for
>> community-built numpy Windows binaries is dire:
>>
>> * We have not so far provided windows wheels on pypi, so `pip install
>> numpy` on Windows will bring you a world of pain;
>> * Until recently we did provide .exe "superpack" installers on
>> sourceforge, but these became increasingly difficult to build and we
>> gave up building them as of the latest (1.10.4) release.
>>
>> Despite this, popularity of Windows wheels on pypi is high.   A few
>> weeks ago, Donald Stufft ran a query for the binary wheels most often
>> downloaded from pypi, for any platform [2] . The top five most
>> downloaded were (n_downloads, name):
>>
>> 6646,
>> numpy-1.10.4-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
>> 5445, cryptography-1.2.1-cp27-none-win_amd64.whl
>> 5243, matplotlib-1.4.0-cp34-none-win32.whl
>> 5241, scikit_learn-0.15.1-cp34-none-win32.whl
>> 4573, pandas-0.17.1-cp27-none-win_amd64.whl
>>
>> So a) the OSX numpy wheel is very popular and b) despite the fact that
>> we don't provide a numpy wheel for Windows, matplotlib, sckit_learn
>> and pandas, that depend on numpy, are the 3rd, 4th and 5th most
>> downloaded wheels as of a few weeks ago.
>>
>> So, there seems to be a large appetite for numpy wheels.
>>
>> Current proposal:
>>
>> I have now built numpy wheels, using the ATLAS blas / lapack library -
>> the build is automatic and reproducible [3].
>>
>> I chose ATLAS to build against, rather than, say OpenBLAS, because
>> we've had some significant worries in the past about the reliability
>> of OpenBLAS, and I thought it better to err on the side of
>> correctness.
>>
>> However, these builds are relatively slow for matrix multiply and
>> other linear algebra routines compared numpy built against OpenBLAS or
>> MKL (which we cannot use because of its license) [4].   In my very
>> crude array test of a dot product and matrix inversion, the ATLAS
>> wheels were 2-3 times slower than MKL.  Other benchmarks on Julia
>> found about the same result for ATLAS vs OpenBLAS on 32-bit bit, but a
>> much bigger difference on 64-bit (for an earlier version of ATLAS than
>> we are currently using) [5].
>>
>> So, our numpy wheels likely to be stable and give correct results, but
>> will be somewhat slow for linear algebra.
>
>
> I would not worry too much about this: at worst, this gives us back the
> situation where we were w/ so-called superpack, which have been successful
> in the past to spread numpy use on windows.
>
> My main worry is whether this locks us into ATLAS  for a long time because
> of package depending on numpy blas/lapack (scipy, scikit learn). I am not
> sure how much this is the case.

You mean the situation where other packages try to find the BLAS /
LAPACK library and link against that?   My impression was that neither
scipy or scikit-learn do that at the moment, but I'm happy to be
corrected.

You'd know better than me about this, but my understanding is that
BLAS / LAPACK has a standard interface that should allow code to run
the same way, regardless of which BLAS / LAPACK library it is linking
to.  So, even if another package is trying to link against the numpy
BLAS, swapping the numpy BLAS library shouldn't cause a problem
(unless the package is trying to link to ATLAS-specific stuff, which
seems a bit unlikely).

Is that right?

Cheers,

Matthew



More information about the NumPy-Discussion mailing list