[Distutils] Handling the binary dependency management problem

[Oscar Benjamin]

On 4 December 2013 20:56, Ralf Gommers <ralf.gommers at gmail.com> wrote:
> On Wed, Dec 4, 2013 at 5:05 PM, Chris Barker - NOAA Federal
> <chris.barker at noaa.gov> wrote:
>>
>> So a lowest common denominator wheel would be very, very, useful.
>>
>> As for what that would be: the superpack is great, but it's been around a
>> while (long while in computer years)
>>
>> How many non-sse machines are there still out there? How many non-sse2?
>
> Hard to tell. Probably <2%, but that's still too much. Some older Athlon XPs
> don't have it for example. And what if someone submits performance
> optimizations (there has been a focus on those recently) to numpy that use
> SSE4 or AVX for example? You don't want to reject those based on the
> limitations of your distribution process.
>
>> And how big is the performance boost anyway?
>
> Large. For a long time we've put a non-SSE installer for numpy on pypi so
> that people would stop complaining that ``easy_install numpy`` didn't work.
> Then there were regular complaints about dot products being an order of
> magnitude slower than Matlab or R.

Yes, I wouldn't want that kind of bad PR getting around about
scientific Python "Python is slower than Matlab" etc.

It seems as if there is a need to extend the pip+wheel+PyPI system
before this can fully work for numpy. I'm sure that the people here
who have been working on all of this would be very interested to know
what kinds of solutions would work best for numpy and related
packages.

You mentioned in another message that a post-install script seems best
to you. I suspect there is a little reluctance to go this way because
one of the goals of the wheel system is to reduce the situation where
users execute arbitrary code from the internet with admin privileges
e.g. "sudo pip install X" will download and run the setup.py from X
with root privileges. Part of the point about wheels is that they
don't need to be "executed" for installation. I know that post-install
scripts are common in .deb and .rpm packages but I think that the use
case there is slightly different as the files are downloaded from
controlled repositories whereas PyPI has no quality assurance.

BTW, how do the distros handle e.g. SSE? My understanding is that they
just strip out all the SSE and related non-portable extensions and
ship generic 686 binaries. My experience is with Ubuntu and I know
they're not very good at handling BLAS with numpy and they don't seem
to be able to compile fftpack as well as Cristoph can.

Perhaps a good near-term plan might be to
1) Add the bdist_wheel command to numpy - which may actually be almost
automatic with new enough setuptools/pip and wheel installed.
2) Upload wheels for OSX to PyPI - for OSX SSE support can be inferred
from OS version which wheels can currently handle.
3) Upload wheels for Windows to somewhere other than PyPI e.g.
SourceForge pending a distribution solution that can detect SSE
support on Windows.

I think it would be good to have a go at wheels even if they're not
fully ready for PyPI (just in case some other issue surfaces in the
process).


Oscar