[Distutils] Handling the binary dependency management problem

[Ralf Gommers]

On Wed, Dec 4, 2013 at 1:54 AM, Donald Stufft <donald at stufft.io> wrote:

>
> On Dec 3, 2013, at 7:36 PM, Oscar Benjamin <oscar.j.benjamin at gmail.com>
> wrote:
>
> > On 3 December 2013 21:13, Donald Stufft <donald at stufft.io> wrote:
> >> I think Wheels are the way forward for Python dependencies. Perhaps not
> for
> >> things like fortran. I hope that the scientific community can start
> >> publishing wheels at least in addition too.
> >
> > The Fortran issue is not that complicated. Very few packages are
> > affected by it. It can easily be fixed with some kind of compatibility
> > tag that can be used by the small number of affected packages.
> >
> >> I don't believe that Conda will gain the mindshare that pip has outside
> of
> >> the scientific community so I hope we don't end up with two systems that
> >> can't interoperate.
> >
> > Maybe conda won't gain mindshare outside the scientific community but
> > wheel really needs to gain mindshare *within* the scientific
> > community. The root of all this is numpy. It is the biggest dependency
> > on PyPI, is hard to build well, and has the Fortran ABI issue. It is
> > used by very many people who wouldn't consider themselves part of the
> > "scientific community". For example matplotlib depends on it. The PyPy
> > devs have decided that it's so crucial to the success of PyPy that
> > numpy's basically being rewritten in their stdlib (along with the C
> > API).
> >
> > A few times I've seen Paul Moore refer to numpy as the "litmus test"
> > for wheels. I actually think that it's more important than that. If
> > wheels are going to fly then there *needs* to be wheels for numpy. As
> > long as there isn't a wheel for numpy then there will be lots of
> > people looking for a non-pip/PyPI solution to their needs.
> >
> > One way of getting the scientific community more on board here would
> > be to offer them some tangible advantages. So rather than saying "oh
> > well scientific use is a special case so they should just use conda or
> > something", the message should be "the wheel system provides solutions
> > to many long-standing problems and is even better than conda in (at
> > least) some ways because it cleanly solves the Fortran ABI issue for
> > example".
> >
> >
> > Oscar
>
> I’d love to get Wheels to the point they are more suitable then they are
> for
> SciPy stuff,


That would indeed be a good step forward. I'm interested to try to help get
to that point for Numpy and Scipy.

I’m not sure what the diff between the current state and what
> they need to be are but if someone spells it out (I’ve only just skimmed
> your last email so perhaps it’s contained in that!) I’ll do the arguing
> for it. I
> just need someone who actually knows what’s needed to advise me :)
>

To start with, the SSE stuff. Numpy and scipy are distributed as
"superpack" installers for Windows containing three full builds: no SSE,
SSE2 and SSE3. Plus a script that runs at install time to check which
version to use. These are built with ``paver bdist_superpack``, see
https://github.com/numpy/numpy/blob/master/pavement.py#L224. The NSIS and
CPU selector scripts are under tools/win32build/.

How do I package those three builds into wheels and get the right one
installed by ``pip install numpy``?

If this is too difficult at the moment, an easier (but much less important
one) would be to get the result of ``paver bdist_wininst_simple`` as a
wheel.

For now I think it's OK that the wheels would just target 32-bit Windows
and python.org compatible Pythons (given that that's all we currently
distribute). Once that works we can look at OS X and 64-bit Windows.

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20131204/948f087b/attachment-0001.html>