[Distutils] What does it mean for Python to "bundle pip"?

[Nick Coghlan]

On 23 August 2013 08:03, Chris Barker - NOAA Federal
<chris.barker at noaa.gov> wrote:
> On Thu, Aug 22, 2013 at 9:37 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>
>> That is essentially possible now.
>>
>> 1. Go to Christoph Gohlke's website and download his bdist_wininst
>> installers for numpy and scipy.
>
> ....
>
> Exactly. And when all this settles down, hopefully Christoph, and
> others, will put up binary wheels and we'll have one stop installing
> that supports virtualenv, and pypi discover of "good enough" binary
> wheels.

Right - this is exactly my ambition. Have pip+wheel+virtualenv provide
a "good enough" out-of-the-box experience, but not necessarily support
a fully optimised experience for any given platform. There are too
many different possible full stack integration technologies for it to
make sense for us to try to supersede them all - instead, I'd like to
provide an 80% cross-platform solution that "plays well enough with
others" to cover the remaining 20%.

The "others" then includes things like zc.buildout, conda, Linux
distro package managers, Microsoft System Centre, automated
configuration management tools, PaaS providers like OpenShift and
Heroku, container technologies like Docker and the underpinnings of
OpenShift. In this space, the goal of the pip+wheel ecosystem will be
to make it possible to either use a command like "pip install
zc.buildout" or "pip install conda" to bootstrap a cross-platform
toolchain, or else to use a platform specific toolchain (like pyp2rpm)
to *consume* the upstream packages and produce nice, policy compliant,
packages automatically.

Managing arbitrary binary dependencies adds a lot of complexity for a
capability that I believe the majority of Python projects don't need.
Even the ones that can use it (like scientific tools) can often
provide a "good enough" fallback option that will fit within the
constraints of the draft metadata 2.0 standards.

> My point is that is may not be wise to try to support the more complex
> builds -- they ARE complex, and trying to suppor it with an auto tool
> is a bit much.

This is where I think "pip install conda" shows a lot of promise as a
good, cross-platform solution, at least in the scientific space.
However, the trade-off it makes is that the hash-based packaging
system means you *always* pin your dependencies when building a
package, with all the downsides that entails (mainly in increasing the
complexity of security updates). It's just that in the scientific
space, easily reproducing a particular software stack will often trump
the desire to make security updates easy to deploy with minimal impact
on other components.

That said, I'm considering the idea of adding a "variant" field to the
compatibility tags for wheel 1.1, along the lines of what Oscar
Benjamin suggested earlier. By default, installers would only find
wheels without a variant defined, but users could opt in to looking
for particular variants. The meaning of the variants field would be
entirely distribution specific. Numpy, for example, could publish:

numpy-1.7.1-cp27-cp22m-win32.whl
numpy-1.7.1-cp27-cp22m-win32-sse.whl
numpy-1.7.1-cp27-cp22m-win32-sse2.whl
numpy-1.7.1-cp27-cp22m-win32-sse3.whl

The only restrictions on the variant tags would be:
1. must be ASCII
2. must not contain '.' or '-' characters

You could even go to the extent of using hashes as variant tags.

> Oscar wrote:
>> And actually 'pip wheel numpy' works fine for me on Windows with MinGW
> installed.
>
> Good start, but the bigger issue is that 'pip install' finds that wheel...
>
> I'm still confused as to the state of all this -- are the tools ready
> for project to start posting wheels so that pip can find them?

As others have noted:

- definite yes for building your own wheels for internal use
(including simple build caching to speed up virtualenv creation)
- qualified yes for publication on PyPI (i.e. "there may still be
rough edges, so don't be too surprised if this still has flaws at this
point, especially on OS X and Linux")

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia