[Distutils] PEP 513: A Platform Tag for Portable Linux Built Distributions Version

[Nathaniel Smith]

Hi all,

I think the PEP is pretty close to ready, but there's one remaining
minor-but-substantive technical point that got mostly skipped in the
previous discussion, and that I'm a little concerned about and wanted
to highlight. This is the mechanism by which a distribution/user can
explicitly specify their manylinux1-compatibility (overriding the
default heuristic rule):

On Mon, Jan 25, 2016 at 1:51 PM, Robert T. McGibbon <rmcgibbo at gmail.com> wrote:
[...]
> To handle the third case, we propose the creation of a file
> ``/etc/python/compatibility.cfg`` in ConfigParser format, with sample
> contents: ::
>
>    [manylinux1]
>    compatible = true
>
> where the supported values for the ``manylinux1.compatible`` entry are the
> same as those supported by the ConfigParser ``getboolean`` method.
>
> The proposed logic for ``pip`` or related tools, then, is:
>
> 0) If ``distutils.util.get_platform()`` does not start with the string
>    ``"linux"``, then assume the current system is not ``manylinux1``
>    compatible.
> 1) If ``/etc/python/compatibility.conf`` exists and contains a
> ``manylinux1``
>    key, then trust that.
> 2) Otherwise, if ``have_glibc_version(2, 5)`` returns true, then assume the
>    current system can handle ``manylinux1`` wheels.
> 3) Otherwise, assume that the current system cannot handle ``manylinux1``
>    wheels.

As stated, this proposal has a few problems:
- it doesn't key off of architecture -- e.g. on Debian a supported
(though weird) configuration would be to have both x86_64 and arm
builds of CPython installed into the same filesystem, and probably it
should be possible to specify the compatibility flag separately for
these two interpreters. If there's just one file in /etc, that isn't
possible.
- it's a single global setting, that applies even to non-global python
installs -- e.g. if I'm using a distribution which has compatible=True
in their /etc/python/compatibility.conf, and then in my home directory
I install a copy of CPython linked against musl-libc, then too bad,
the configuration file still applies and I'm going to get broken
wheels installed into my home directory environment.

Basically, in general, manylinux1-compatibility is not a property of a
system as a whole; it's a property of a single python interpreter
environment within that system. (I guess one can also imagine it
having different values within the same distro for a build of
python2.7 versus python3.4 versus pypy.)

So what would should we do? Some options that occur to me; none of
them obviously right :-):

- Make the configuration file lookup rules more elaborate -- e.g.,
look for a file called $WHEEL_ABI_TAG.cfg, and check first next to the
interpreter itself (re-using the search rule for pyvenv.cfg defined in
PEP 405) and then in /etc if that fails.

- Since this is a property of a particular python environment, bake it
into the environment itself, e.g. as a special variable
sys._manylinux1 that distributors can patch into place if they need to
override this.

- Just forget about the whole override concept for now; switch to
unconditionally checking for glibc version as the only rule, and wait
for someone to run into a case where they need the override feature
before adding it back.

Thoughts? I'd particularly like to hear from anyone associated with
the distros here, since this feature is intended to try and make them
happy :-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org