[Distutils] Handling the binary dependency management problem

[Ralf Gommers]

On Thu, Dec 5, 2013 at 10:12 PM, Oscar Benjamin
<oscar.j.benjamin at gmail.com>wrote:

> On 4 December 2013 20:56, Ralf Gommers <ralf.gommers at gmail.com> wrote:
> > On Wed, Dec 4, 2013 at 5:05 PM, Chris Barker - NOAA Federal
> > <chris.barker at noaa.gov> wrote:
> >>
> >> So a lowest common denominator wheel would be very, very, useful.
> >>
> >> As for what that would be: the superpack is great, but it's been around
> a
> >> while (long while in computer years)
> >>
> >> How many non-sse machines are there still out there? How many non-sse2?
> >
> > Hard to tell. Probably <2%, but that's still too much. Some older Athlon
> XPs
> > don't have it for example. And what if someone submits performance
> > optimizations (there has been a focus on those recently) to numpy that
> use
> > SSE4 or AVX for example? You don't want to reject those based on the
> > limitations of your distribution process.
> >
> >> And how big is the performance boost anyway?
> >
> > Large. For a long time we've put a non-SSE installer for numpy on pypi so
> > that people would stop complaining that ``easy_install numpy`` didn't
> work.
> > Then there were regular complaints about dot products being an order of
> > magnitude slower than Matlab or R.
>
> Yes, I wouldn't want that kind of bad PR getting around about
> scientific Python "Python is slower than Matlab" etc.
>
> It seems as if there is a need to extend the pip+wheel+PyPI system
> before this can fully work for numpy. I'm sure that the people here
> who have been working on all of this would be very interested to know
> what kinds of solutions would work best for numpy and related
> packages.
>
> You mentioned in another message that a post-install script seems best
> to you. I suspect there is a little reluctance to go this way because
> one of the goals of the wheel system is to reduce the situation where
> users execute arbitrary code from the internet with admin privileges
> e.g. "sudo pip install X" will download and run the setup.py from X
> with root privileges. Part of the point about wheels is that they
> don't need to be "executed" for installation. I know that post-install
> scripts are common in .deb and .rpm packages but I think that the use
> case there is slightly different as the files are downloaded from
> controlled repositories whereas PyPI has no quality assurance.
>

I don't think it's avoidable - anything that is transparant to the user
will have to execute code. The multiwheel idea of Nick looks good to me.


> BTW, how do the distros handle e.g. SSE?
>

I don't know exactly to be honest.


> My understanding is that they
> just strip out all the SSE and related non-portable extensions and
> ship generic 686 binaries. My experience is with Ubuntu and I know
> they're not very good at handling BLAS with numpy and they don't seem
> to be able to compile fftpack as well as Cristoph can.
>
> Perhaps a good near-term plan might be to
> 1) Add the bdist_wheel command to numpy - which may actually be almost
> automatic with new enough setuptools/pip and wheel installed.
> 2) Upload wheels for OSX to PyPI - for OSX SSE support can be inferred
> from OS version which wheels can currently handle.
> 3) Upload wheels for Windows to somewhere other than PyPI e.g.
> SourceForge pending a distribution solution that can detect SSE
> support on Windows.
>

That's a reasonable plan. I have an OS X wheel already, which required only
a minor change to numpy's setup.py.


> I think it would be good to have a go at wheels even if they're not
> fully ready for PyPI (just in case some other issue surfaces in the
> process).
>

Agreed.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131205/ffdd3112/attachment-0001.html>