On 4 December 2013 20:56, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Wed, Dec 4, 2013 at 5:05 PM, Chris Barker - NOAA Federal <chris.barker@noaa.gov> wrote:
So a lowest common denominator wheel would be very, very, useful.
As for what that would be: the superpack is great, but it's been around a while (long while in computer years)
How many non-sse machines are there still out there? How many non-sse2?
Hard to tell. Probably <2%, but that's still too much. Some older Athlon XPs don't have it for example. And what if someone submits performance optimizations (there has been a focus on those recently) to numpy that use SSE4 or AVX for example? You don't want to reject those based on the limitations of your distribution process.
And how big is the performance boost anyway?
Large. For a long time we've put a non-SSE installer for numpy on pypi so that people would stop complaining that ``easy_install numpy`` didn't work. Then there were regular complaints about dot products being an order of magnitude slower than Matlab or R.
Yes, I wouldn't want that kind of bad PR getting around about scientific Python "Python is slower than Matlab" etc. It seems as if there is a need to extend the pip+wheel+PyPI system before this can fully work for numpy. I'm sure that the people here who have been working on all of this would be very interested to know what kinds of solutions would work best for numpy and related packages. You mentioned in another message that a post-install script seems best to you. I suspect there is a little reluctance to go this way because one of the goals of the wheel system is to reduce the situation where users execute arbitrary code from the internet with admin privileges e.g. "sudo pip install X" will download and run the setup.py from X with root privileges. Part of the point about wheels is that they don't need to be "executed" for installation. I know that post-install scripts are common in .deb and .rpm packages but I think that the use case there is slightly different as the files are downloaded from controlled repositories whereas PyPI has no quality assurance. BTW, how do the distros handle e.g. SSE? My understanding is that they just strip out all the SSE and related non-portable extensions and ship generic 686 binaries. My experience is with Ubuntu and I know they're not very good at handling BLAS with numpy and they don't seem to be able to compile fftpack as well as Cristoph can. Perhaps a good near-term plan might be to 1) Add the bdist_wheel command to numpy - which may actually be almost automatic with new enough setuptools/pip and wheel installed. 2) Upload wheels for OSX to PyPI - for OSX SSE support can be inferred from OS version which wheels can currently handle. 3) Upload wheels for Windows to somewhere other than PyPI e.g. SourceForge pending a distribution solution that can detect SSE support on Windows. I think it would be good to have a go at wheels even if they're not fully ready for PyPI (just in case some other issue surfaces in the process). Oscar