[Distutils] Binary dependency management, round 2 :)

[Nick Coghlan]

On 7 December 2013 03:44, Chris Barker <chris.barker at noaa.gov> wrote:
> On Fri, Dec 6, 2013 at 6:22 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> >> I created a draft of this new section at
>> >>
>> >> https://bitbucket.org/pypa/python-packaging-user-guide/pull-request/12/recommendations-for-numpy-et-al/diff
>
> looks good, thanks!

With some additional suggestions from Oscar Benjamin and Ned Deilly,
this is now live on the user guide:

https://python-packaging-user-guide.readthedocs.org/en/latest/additional.html#installing-scientific-packages

> ONe note:
>
> """
> In particular, bootstrapping conda via ``pip install conda`` and then
> running the ``conda init`` command provides access to all of the pre-built
> binaries that Continuum Analytics have created for the free version of the
> Anaconda distribution.
>  """
>
> I've been chatting off list with Travis, and while it does appear that
> Anaconda is more compatibly with the python.org installers than I had
> thought, there are still some pretty significant "rough edges", particularly
> with the OS-X build. But it does look pretty promising.
>
> Travis pointed out that I had been pretty critical of endorsing conda in
> this thread. He is right. That came from two things:
>
> 1) years of frustration with the mess of python packaging, that I thought we
> were finally resolving with binary wheels.

We are, but I think there's still a great deal of value in *curated*
platforms. PyPI works rather well for a relatively anarchic
free-for-all, but we aren't going to completely get rid of the case of
somebody depending on another package that has a binary extension, but
doesn't publish a wheel file until we have an integrated build farm
that generates wheels from uploaded sdists automatically (such a thing
is definitely on the long term wish list, but there are a great many
higher priority problems ahead of it on the todo list).

> 2) recent bad experience with Anaconda and teaching python to newbies.
>
> I'm pretty sure we all want  "one way to do it" -- i.e. we can just tell
> people to install a python.org build, then  use pip install to get
> everything else, than may simply not be practical, and apparently we're not
> as close as I thought.

If we can resolve the numpy situation (either with OpenBLAS, or by
nominating ATLAS SSE2 as "good enough for now"), then we'll be in a
pretty good place for Windows and Mac OS X. Other *nix systems will
still be a challenge, since there may be more C ABI variations to
cover, and the current definition of the "platform" tag doesn't do a
good job of handling variation across Linux distros.

> But as I think back over the last few years, I realize that I've been
> recommending the python.org binaries across the board, because it is the
> basis of a number of different package approaches. So I can tell my
> students:
>
>  - try pip  install
>
>  - if that doesn't work, look in the Gohlke repo
>
>  - if that doesn't work, look for a binary on the package web page.
>
>  - if that doesn't work, follow the build instructions on the package web
> page
>
> And while it would be great if pip install always worked, this really isn't
> so bad, and it's not so bad because, in fact, most of the maintainers of
> complex packages have been targeting python.org binaries for years.
>
> So if we can't have wheels for everything, we can do pretty well if the
> other options (e.g. conda)  are python.org compatible. That's not quite the
> case for Anaconda now, but it's not that far off, and Travis is interested
> in making it better.

The Anaconda folks just pushed a new release today which should fix
the binary extension compatibility issue for 3.3 on Windows. I'm not
aware of the situation on Mac OS X, but they were quite responsive to
the ticket Paul filed about the compatibility problem on Windows.

> I'm not going to have time to poke at this myself for at least a few weeks,
> but at some point, maybe we can, for instance, try to convert the wxPython
> binaries to a conda package, or just figure out a post-install hack that
> will let it work with Anaconda (the current wxPython binaries for OS-X
> already have *.pth file magic to let both the python.org and Apple binaries
> use the same package).

I believe the conda folks are also working on automatically falling
back to pip - if conda doesn't find a package in the conda repos, use
pip to go hunting on PyPI instead. With the PyPI package count
outnumbering the curated Anaconda package list by a few orders of
magnitude (~34k vs a hundred or so) such an approach should resolve a
lot of "conda install doesn't work for package <X>" issues :)

> So nice work all -- it seem this is not getting as ugly as I feared.

Yeah, I see a great deal of potential benefit to users in conda as a
"cross platform platform", so from an upstream tooling perspective, I
think it makes sense for us to treat it similarly to how we treat
other platforms. The main difference is that there's additional
integration in the other direction in the conda case.

I think these threads have also made it clear that there's a docs
issue with sysconfig and the "data" directory, but I'll start a
separate thread about that.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia