[Distutils] Handling the binary dependency management problem

Ralf Gommers ralf.gommers at gmail.com
Wed Dec 4 22:13:52 CET 2013


On Wed, Dec 4, 2013 at 9:13 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 4 December 2013 07:40, Ralf Gommers <ralf.gommers at gmail.com> wrote:
> >> I’m not sure what the diff between the current state and what
> >> they need to be are but if someone spells it out (I’ve only just skimmed
> >> your last email so perhaps it’s contained in that!) I’ll do the arguing
> >> for it. I
> >> just need someone who actually knows what’s needed to advise me :)
> >
> >
> > To start with, the SSE stuff. Numpy and scipy are distributed as
> "superpack"
> > installers for Windows containing three full builds: no SSE, SSE2 and
> SSE3.
> > Plus a script that runs at install time to check which version to use.
> These
> > are built with ``paver bdist_superpack``, see
> > https://github.com/numpy/numpy/blob/master/pavement.py#L224. The NSIS
> and
> > CPU selector scripts are under tools/win32build/.
> >
> > How do I package those three builds into wheels and get the right one
> > installed by ``pip install numpy``?
>
> I think that needs a compatibility tag. Certainly it isn't immediately
> soluble now.
>
> Could you confirm how the correct one of the 3 builds is selected
> (i.e., what the code is to detect which one is appropriate)? I could
> look into what options we have here.
>

The stuff under tools/win32build I mentioned above. Specifically:
https://github.com/numpy/numpy/blob/master/tools/win32build/cpuid/cpuid.c


> > If this is too difficult at the moment, an easier (but much less
> important
> > one) would be to get the result of ``paver bdist_wininst_simple`` as a
> > wheel.
>
> That I will certainly look into. Simple answer is "wheel convert
> <wininst>". But maybe it would be worth adding a "paver bdist_wheel"
> command. That should be doable in the same wahy setuptools added a
> bdist_wheel command.
>
> > For now I think it's OK that the wheels would just target 32-bit Windows
> and
> > python.org compatible Pythons (given that that's all we currently
> > distribute). Once that works we can look at OS X and 64-bit Windows.
>
> Ignoring the SSE issue, I believe that simply wheel converting
> Christoph Gohlke's repository gives you that right now. The only
> issues there are (1) the MKL license limitation, (2) hosting, and (3)
> whether Christoph would be OK with doing this (he goes to lengths on
> his site to prevent spidering his installers).
>

Besides the issues you mention, the problem is that it creates a single
point of failure. I really appreciate everything Christoph does, but it's
not appropriate as the default way to provide binary releases for a large
number of projects. There needs to be a reproducible way that the devs of
each project can build wheels - this includes the right metadata, but
ideally also a good way to reproduce the whole build environment including
compilers, blas/lapack implementations, dependencies etc. The latter part
is probably out of scope for this list, but is discussed right now on the
numfocus list.


> I genuinely believe that "a schientific stack for non-scientists" is
> trivially solved in this way.


That would be nice, but no. The only thing you'd have achieved is to take a
curated stack of .exe installers and converted it to the same stack of
wheels. Which is nice and a step forward, but doesn't change much in the
bigger picture. The problem is certainly nontrivial.

Ralf


> For scientists, of course, we'd need to
> look deeper, but having a base to start from would be great.
>
> Paul
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131204/73e8931d/attachment.html>


More information about the Distutils-SIG mailing list