[Distutils] draft PEP: manylinux1

[Nathaniel Smith]

On Thu, Jan 21, 2016 at 2:22 PM, Nate Coraor <nate at bx.psu.edu> wrote:
> Nathaniel, Robert, I'm really excited to see how quickly you're making
> progress. A few comments below as I haven't had a chance to catch up on the
> day's discussion:
>
> On Wed, Jan 20, 2016 at 10:55 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>> Building on the compability lessons learned from these companies, we thus
>> define a baseline ``manylinux1`` platform tag for use by binary Python
>> wheels, and introduce the implementation of preliminary tools to aid in
>> the
>> construction of these ``manylinux1`` wheels.
>
>
> Just a standards question: does this still require an update to PEP 425, or
> would the definition of the  manylinux1 platform here supersede that section
> of 425?

I guess this is a pure process question, so I'll defer to Nick...

>> * Eventually, in the future, there may exist distributions that break
>>   compatibility with this profile
>
>
>>
>> To handle the third case, we propose the creation of a file
>> ``/etc/python/compatibility.cfg`` in ConfigParser format, with sample
>> contents: ::
>>
>>    [manylinux1]
>>    compatible = true
>>
>> where the supported values for the ``manylinux1.compatible`` entry are the
>> same as those supported by the ConfigParser ``getboolean`` method.
>
>
> Could this instead use the more powerful json-based syntax proposed by Nick
> here:
>
>     https://mail.python.org/pipermail/distutils-sig/2015-July/026617.html
>
> I have already implemented support for this in pip and wheel.

Totally happy to change the compatibility.cfg stuff -- the version in
the PEP was written in about 5 minutes in hopes of sparking discussion
:-).

Some other questions:
1) is this going to work for multi-arch (binaries for multiple cpu
architectures sharing a single /etc)? Multiple interpreters? I guess
the advantage of Nick's design is that it's scoped by the value of
distutils.util.get_platform(), so multi-arch installs could have
different values -- a distro could declare that their x86-64 python
builds are manylinux1 compatible but their i386 python builds aren't.
Maybe it would be even better if the files were
/etc/python/binary-compatibility/linux_x86_64.cfg etc., so that the
different .cfg files could be shipped in each architecture's package
without colliding. OTOH I don't know if any of this is very useful in
practice.
2) in theory one could imaging overriding this on a per-venv,
per-user, or per-system level; which of these are useful to support?
(Per-system seems like the most obvious, since an important use case
will be distros setting this flag on behalf of their users.)

There is one feature that I do think is important in the PEP 513
draft, and that Nick's suggestion from July doesn't have: in the PEP
513 design, the manylinux1 flag can be true, false, or unspecified,
and this is independent of other compatibility settings; in Nick's
proposal there's an exhaustive list of all compatible tags, and
everything not on that list is assumed to be incompatible. Where this
matters is when we want to release manylinux2. At this point we'll
want pip to use some autodetection logic on old distros that were
released before manylinux2, while respecting the compatibility flag on
newer distros that do know about manylinux2. This requires a tri-state
setting with "not specified" as a valid value.

[...]
>> Rejected Alternatives
>> =====================
>>
>> One alternative would be to provide separate platform tags for each Linux
>> distribution (and each version thereof), e.g. ``RHEL6``, ``ubuntu14_10``,
>> ``debian_jessie``, etc. Nothing in this proposal rules out the possibility
>> of
>> adding such platform tags in the future, or of further extensions to wheel
>> metadata that would allow wheels to declare dependencies on external
>> system-installed packages. However, such extensions would require
>> substantially
>> more work than this proposal, and still might not be appreciated by
>> package
>> developers who would prefer not to have to maintain multiple build
>> environments
>> and build multiple wheels in order to cover all the common Linux
>> distributions.
>> Therefore we consider such proposals to be out-of-scope for this PEP.
>
>
> ;)
>
> For anyone who's interested, the next release of Galaxy (a popular
> bioinformatics framework for running tools, workflows, etc.), due next week,
> will ship with our modified pip that includes support for
> distro/version-specific platform tags in wheels. All but one of our
> dependent package's wheels are built with the generic `linux_x86_64` tag on
> Debian Squeeze and will work with most distros, though, so we're basically
> doing a "loose" version of manylinux1. Only our psycopg2 wheels are built
> per-distro/version. I'm happy to see a more rigid definition for what we're
> doing with the "generic" ones, this is certainly necessary should support
> for generic Linux wheels ever be allowed into PyPI. manylinux1 and this PEP
> seem to me like the right idea to do this.
>
> We build these distro/version wheels using a modified wheel and the
> aforementioned docker-based build system, called Starforge. Here's where all
> of it lives:
>
> Galaxy: https://github.com/galaxyproject/galaxy
> Starforge: https://github.com/galaxyproject/starforge
> "Galaxy pip": https://github.com/natefoo/pip/tree/linux-wheels
> "Galaxy wheel": https://bitbucket.org/natefoo/wheel
>
> I say all of this (again) because I think there's still a place for the work
> we've done even if manylinux1 is accepted. Both manylinux1 *and* more
> specific distro/version platform tags can happily coexist. So I want to
> publicize the fact that this work is already done and in use (although I
> know there are some things that would need to be done before it could be
> upstreamed).
>
> Basically I'm trying to determine whether there's any interest in this from
> the pip and PyPI developers. If so, I'd be happy to write a PEP that
> complements the one written by Robert and Nathaniel so that both types of
> platform tags and wheels using them can be supported.

I don't have anything in particular to say about this part, except
that it's really cool :-)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org