On Fri, 30 Nov 2018 at 08:12, Nathaniel Smith <njs@pobox.com> wrote:
Hi all,
The manylinux1 -> manylinux2010 transition has turned out to be very difficult. Timeline so far:
March 2017: CentOS 5 went EOL April 2018: PEP 517 accepted May 2018: support for manylinux2010 lands in warehouse November 2018: support lands in auditwheel, and pip master December 2018: 21 months after CentOS 5 EOL, wwee still don't have an official build environment, or support in a pip release
We'll get through this, but it's been super painful and maybe we can change things somehow so it will suck less next time.
We don't have anything like this pain on Windows or macOS. We never have to update pip, warehouse, etc., after those OSes hit EOLs. Why not?
As a Windows user who doesn't understand the whole Linux ABI situation[1], I can't answer that. But I do think that the goal should be that we *don't* need changes to pip and Warehouse in order to keep Linux wheels current. Whether that's done by not needing new tags (the way Windows does it) or by having a general "pattern" of tags that needs no maintenance (the way macOS does it) I don't know.
On Windows, we have just two tags: "win32" and "win_amd64". These are defined to mean something like "this wheel will run on any recent-ish Windows system". So the meaning of the tag actually changes over time: it used to be that if a wheel said it ran on win32, then that meant it would work on winxp, but since winxp hit EOL people started uploading "win32" wheels that don't work on winxp, and that's worked fine.
On macOS, the tags look like "macosx_10_9_x86_64". So here we have the OS version embedded in the tag. This means that we do occasionally switch which tags we're using, kind of like how manylinux1 -> manylinux2010 is intended to work. But, unlike for the manylinux tags, defining a new macosx tag is totally trivial: every time a new OS version is released, the tag springs into existence without any human intervention. Warehouse already accepts uploads with this tag; pip already knows which systems can install wheels with this tag, etc.
Can we take any inspiration from this for manylinux?
Only Linux users can really answer this. But what I will say is that on Windows, anything other than the core system libraries must be bundled in the wheel (so, for example, Pillow bundles the various image handling DLLs). Manylinux (as I understand it) does a certain amount of this, but expects dynamic linking for a much wider set of libraries. Maybe that reflects the same sort of mindset that results in Linux distros "debundling" tools like pip that vendor their dependencies. I'm not going to try to judge whether the Linux or the Windows approach is "right", but I'd be surprised if manylinux can take much inspiration from the Windows approach without confronting this difference in philosophy. Paul [1] I certainly don't want to spark any sort of flamewar here, but I do feel a certain wry amusement that the term "DLL Hell" was invented as a criticism of library management practices on Windows, and yet in this context, library management on Windows is pretty much a non-problem, and it's Linux (that prided itself on avoiding DLL hell at the time) that is now struggling with library versioning complexity ;-)