On Fri, 30 Nov 2018 at 18:12, Nathaniel Smith <njs@pobox.com> wrote:
We could do the Windows thing, and have a plain "manylinux" tag that means "any recent-ish glibc-based Linux". Today it would be defined to be "any distro newer than CentOS 6". When CentOS 6 goes out of service, we could tweak the definition to be "any distro newer than CentOS 7". Most parts of the toolchain wouldn't need to be updated, though, because the tag wouldn't change, and by assumption, enforcement wouldn't really be needed, because the only people who could break would be ones running on unsupported platforms. Just like happens on Windows.
The reason this approach works for Windows is because *CPython* defines the target Windows ABI version - if you don't use the right target ABI, your extension module won't even link to the CPython DLL. So here, we're taking advantage of Microsoft's strict ABI management policy, by way of CPython.
We could do the macOS thing, and have a "manylinux_${glibc version}" tag that means "this package works on any Linux using glibc newer than ${glibc version}". We're already using this as our heuristic to handle the current manylinux profiles, so e.g. manylinux1 is effectively equivalent to manylinux_2_5, and manylinux2010 will be equivalent to manylinux_2_12. That way we'd define the manylinux tags once, get support into pip and warehouse and auditwheel once, and then in the future the only thing that would have to change to support new distro releases or new architectures would be to set up a proper build environment.
This approach works for Mac OS X because Apple have just as strict an approach to ABI management as Microsoft do, and the OS version specifies a lot of details about a broad range of operating system interfaces, which are then given strict compatibility guarantees.
What do y'all think?
I don't think we can get away from *something* specifying exactly what can be assumed to be present in a given manylinux variant, since the distros don't define any useful form of cross-distro ABI compatibility, and CPython doesn't nominate a target Linux ABI either. However, I like the design concept of making it so that auditwheel is the only project that has to change in order to define a new revision of manylinux, which means encoding the heuristic check that installers should use directly into the platform compatibility tag rather than defining it as a lookup table that needs to be updated in every affected tool whenever a new version is defined. From my view, the most promising path towards achieving that would be to go with Brett's suggestion of "manylinux_{libc_variant}_{libc_version}", such that we get "manylinux_glibc_2_17" as the next edition rather than "manylinux2014" (assuming the trend of using RHEL/CentOS releases as the baseline continues). If folks prove keen to start using the new cheaper aarch64 cloud systems, there may also be demand for a "manylinux_glibc_2_27" (compatible with Ubuntu 18.04 and RHEL/CentOS 8). While having both "many" and "glibc" in the name may seem redundant initially, I think it's worth having them both for two reasons: 1. Even if new manylinux tags are being defined as part of the auditwheel documentation rather than in a PEP, I'd still expect them to allow the generated wheel archives to dynamically link against more than just glibc. Accordingly, there will be some platforms that pass the installer's heuristic check, but nevertheless still have compatibility problems with prebuilt wheel archives. 2. PEP 513 defined a ctypes based algorithm for checking the glibc version, and a similar heuristic could be defined for other libc implementations (most notably musl, thanks to Docker/Alpine). If we're going to go through the process of switching to a different naming scheme, we may as well provide a path towards resolving that issue as well, rather than further entrenching manylinux as being glibc-only. Cheers, Nick. P.S. Paul asked how we can have manylinux tags without updating PEP 425 to include them, and the answer is that the actual compatibility tag spec is at https://packaging.python.org/specifications/platform-compatibility-tags/ and that references PEP 513 (manylinux1) and PEP 571 (manylinux2010) in addition to PEP 425. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia