[Distutils] Binary dependency management, round 2 :)

[Nick Coghlan]

There was some really good feedback in the binary dependency thread,
but it ended up going off on a few different tangents. Rather than
expecting people to read the whole thing, I figured I'd try to come up
with a summary of where it has gone so far, and where we might want to
take it from here.

Given the diversity of topics, this should arguably be multiple new
threads, but that approach has its own problems, since some of these
topics are at least somewhat interrelated. In several cases, some of
the discussion could possible move to the tracker issues I created :)

== Regarding documentation ==

One of the things I got from the thread is that we don't currently
have a clear overview published anywhere of *why* people use binary
extensions. The distutils docs and both dated and focus solely on the
mechanics, without discussing either use cases, or the benefits and
limitations of using them.

We also don't have a good location to describe the approach of
statically linking or bundling additional libraries into "platlib" to
avoid issues with external dependencies and incompatible ABI changes
when distributing wheel files.

This seems like something that would be suitably covered as an
"Advanced Topic" in the packaging user's guide, so I filed an issue
with some more specific ideas:
https://bitbucket.org/pypa/python-packaging-user-guide/issue/36/add-a-section-that-covers-binary


== Regarding conda ==

In terms of providing an answer to the question "Where does conda fit
in the scheme of packaging tools?", my conclusion from the thread is
that once a couple of security related issues are fixed (think PyPI
before the rubygems.org compromise for the current state of conda's
security model), and once the Python 3.3 compatibility issue is
addressed on Windows, it would be reasonable to recommend it as one of
the current options for getting hold of pre-built versions of the
scientific Python stack.

I think this is important enough to warrant a "NumPy and the
Scientific Python stack" section in the user guide (with Linux distro
packages, Windows installers and conda all discussed as options):

https://bitbucket.org/pypa/python-packaging-user-guide/issue/37/add-a-dedicated-numpy-and-the-scientific


== Regarding alternative index servers ==

Something I believe conda supports is the idea of installer
configuration settings that are specific to a particular virtual
environment (I can't find specific docs confirming that capability,
but I certainly got that impression somewhere along the line).

At the moment, it isn't straightforward to tell pip "when in this
virtual environment, use these additional settings", but I believe
such a feature could prove useful in dealing with some of the thornier
binary compatibility problems. In particular, it would be good to be
able to lock an environment to a particular custom index server that
it will use instead of defaulting to PyPI.

I've posted this idea to the pip issue tracker:
https://github.com/pypa/pip/issues/1362


== Regarding NumPy, build variants and API/ABI consistency ==

My current reading of the NumPy situation is that it boils down to
needing two additional features:
- a richer platform tagging mechanism (which we need for *nix systems anyway)
- a way to ensure internal consistency of the installed *builds* in an
environment, not just the abstract dependencies

I've opened a wheel format definition issue for the first problem:
https://bitbucket.org/pypa/pypi-metadata-formats/issue/15/enhance-the-platform-tag-definition-for

I've posted a metadata 2.0 standard extension proposal for the latter:
https://bitbucket.org/pypa/pypi-metadata-formats/issue/14/add-a-metadata-based-consistency-checking
(the alternative index server idea is also relevant, since PyPI
wouldn't support hosting multiple variants with the same wheel level
compatibility tags)


== Regarding custom installation directories ==

This technically came up in the cobblerd thread (regarding installing
scripts to /usr/sbin instead of /usr/bin), but I believe it may also
be relevant to the problem of shipping external libraries inside
wheels, static data files for applications, etc.

It's a little underspecified in PEP 427, but the way the wheel format
currently handles installation to paths other than purelib and platlib
(or to install to both of those as part of the same wheel) is to use
the sysconfig scheme names as subdirectories within the wheel's .data
directory. This approach is great for making it easy to build
well-behaved cross platform wheels that play nice with virtual
environments, but allowing a "just put it here" escape clause could
potentially be a useful approach for platform specific wheels
(especially on *nix systems that use the Filesystem Hierarchy
Standard).

I've posted this idea to the metadata format issue tracker:
https://bitbucket.org/pypa/pypi-metadata-formats/issue/13/add-a-new-subdirectory-to-allow-wheels-to

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia