On Wed, Dec 4, 2013 at 9:13 AM, Paul Moore <p.f.moore@gmail.com> wrote:

On 4 December 2013 07:40, Ralf Gommers <ralf.gommers@gmail.com> wrote:
>> I’m not sure what the diff between the current state and what
>> they need to be are but if someone spells it out (I’ve only just skimmed
>> your last email so perhaps it’s contained in that!) I’ll do the arguing
>> for it. I
>> just need someone who actually knows what’s needed to advise me :)
>
>
> To start with, the SSE stuff. Numpy and scipy are distributed as "superpack"
> installers for Windows containing three full builds: no SSE, SSE2 and SSE3.
> Plus a script that runs at install time to check which version to use. These
> are built with ``paver bdist_superpack``, see
> https://github.com/numpy/numpy/blob/master/pavement.py#L224. The NSIS and
> CPU selector scripts are under tools/win32build/.
>
> How do I package those three builds into wheels and get the right one
> installed by ``pip install numpy``?

I think that needs a compatibility tag. Certainly it isn't immediately
soluble now.

Could you confirm how the correct one of the 3 builds is selected
(i.e., what the code is to detect which one is appropriate)? I could
look into what options we have here.

The stuff under tools/win32build I mentioned above. Specifically: https://github.com/numpy/numpy/blob/master/tools/win32build/cpuid/cpuid.c

> If this is too difficult at the moment, an easier (but much less important
> one) would be to get the result of ``paver bdist_wininst_simple`` as a
> wheel.

That I will certainly look into. Simple answer is "wheel convert
<wininst>". But maybe it would be worth adding a "paver bdist_wheel"
command. That should be doable in the same wahy setuptools added a
bdist_wheel command.

> For now I think it's OK that the wheels would just target 32-bit Windows and
> python.org compatible Pythons (given that that's all we currently
> distribute). Once that works we can look at OS X and 64-bit Windows.

Ignoring the SSE issue, I believe that simply wheel converting
Christoph Gohlke's repository gives you that right now. The only
issues there are (1) the MKL license limitation, (2) hosting, and (3)
whether Christoph would be OK with doing this (he goes to lengths on
his site to prevent spidering his installers).

Besides the issues you mention, the problem is that it creates a single point of failure. I really appreciate everything Christoph does, but it's not appropriate as the default way to provide binary releases for a large number of projects. There needs to be a reproducible way that the devs of each project can build wheels - this includes the right metadata, but ideally also a good way to reproduce the whole build environment including compilers, blas/lapack implementations, dependencies etc. The latter part is probably out of scope for this list, but is discussed right now on the numfocus list.

I genuinely believe that "a schientific stack for non-scientists" is
trivially solved in this way.

That would be nice, but no. The only thing you'd have achieved is to take a curated stack of .exe installers and converted it to the same stack of wheels. Which is nice and a step forward, but doesn't change much in the bigger picture. The problem is certainly nontrivial.

Ralf

For scientists, of course, we'd need to
look deeper, but having a base to start from would be great.

Paul