[Distutils] draft PEP: manylinux1

[Nick Coghlan]

On 24 January 2016 at 12:31, Robert T. McGibbon <rmcgibbo at gmail.com> wrote:
> On Sat, Jan 23, 2016 at 6:19 PM, Chris Barker <chris.barker at noaa.gov> wrote:
>>
>> 1)  each package that needs a third partly lib statically links it in.
>> 2)  each package that needs a third partly lib provides it, linked with a
>> relative path (IIUC, that's how most Windows packages are done.
>> 3) We establish some standard for providing binary libs as wheels, so that
>> other packages can depend on them and link to them.
>
> In my view, all of these are valid options. I think much of this will need
> to be worked out by the communities -- especially if individual packages and
> subcommunities decide to take the option (3) approach. I hope this PEP will
> enable the communities involved in OpenGIS, audio processing, image
> processing, etc to work out the solutions that work for them and their
> users.
>
> Perhaps one thing that is missing from the PEP is an explicit statement that
> option (3) is compatible with the manylinux1 tag -- bundling is a valid
> solution, but it's not the *only* solution.

I've long resisted the notion of defining our own cross-distro
platform ABI, but the Docker build environment that was put together
for the manylinux project has made me realise that doing that may not
be as hellish in a post-Docker world as it would have been in a
pre-Docker world. (Since we can go with the specification + reference
implementation approach that CPython has used so successfully for so
long, rather than having to have the build environment and ABI
specification be entirely exhaustive).

On Windows and Mac OS X, our binary compatibility policies for wheel
files are actually pretty loose - it's "be binary compatible with the
python.org builds for that platform, including linking against the
appropriate C standard library", and that's about it. Upgrades to
those ABIs are then driven by CPython switching to newer base
compatibility levels (dropping end-of-life versions on the Windows
side [1], and updating to new deployment target macros on the Mac OS X
side). Folks with external dependencies either bundle them, skip
publishing wheel files, or just let them fail at import time if the
external dependency is missing. (Neither platform has an anti-bundling
culture, though, so I assume a lot of folks go with the first option
over the last one)

If the aim is to bring Linux wheel support in line with Windows and
Mac OS X, then rather than defining a *new* compatibility tag (which
would require new pip clients to process), perhaps we could instead
adopt a similarly loose policy on what the existing generic "linux"
tag means as we have for Windows and Mac OS X: it could just mean
wheel files that are binary compatible with the Python binaries in the
"manylinux" build environment. The difference would then just be that
the target Linux ABI would be defined by PyPA and the manylinux
developers, rather than by python-dev.

In terms of the concerns regarding the age of gcc needed to target
CentOS 5.11, it would be good to know just what nominating CentOS 6.x
as the baseline ABI instead would buy us - CentOS 5 is going on 9
years old (released 2007) and stopped receiving full updates back in
2014 [2], while RHEL/CentOS 6 is just over 5 years old and has another
year of full updates left. The CentOS 6 ABI should still be old enough
to be compatible with the Debian 6 ABI (current stable is Debian 8),
as well as the Ubuntu 12.04 LTS ABI (Ubuntu 16.04 LTS is due out in a
few months).

Cheers,
Nick.

[1] https://www.python.org/dev/peps/pep-0011/#microsoft-windows
[2] https://wiki.centos.org/About/Product

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia