On Fri, Mar 30, 2018, 09:13 Trishank Kuppusamy, < trishank.kuppusamy@datadoghq.com> wrote:

Hi Nick,

Long time no see --- hope everything is well on your side :)

First of all, thanks Mark, Thomas, and everyone else for your hard work on manylinux2010 --- I'm excited about this release, as I'm sure many others are!

This may be the wrong thread to discuss this,

It is, so I've forked the conversation 😉 and

I haven't looked too deeply into it, but are there plans to support container-specific Linux distros like Alpine that use musl instead of glibc?

Are there other container-specific distros other than Alpine that people use widely? I'm guessing the answer is "no," but I'm curious to hear what others think

on the subject.

The discussion of distro-specific wheels has been brought up. I believe they are technically supported if you have your own index server, but PyPI doesn't, I believe, to keep an explosion of wheels people can't widely use down. Currently, if you want to cover all formats PyPI supports you need to produce 5 wheels. Adding a musl-based wheel format would bump that to 7 (as I assume musl supports 23 and 64-bit OSs). For me, the questions of whether to support musl-based wheels on PyPI is will there be enough users to justify the cost to all project maintainers to produce 2 more wheels per release and what would the manylinux-equivalent spec look like (since it would have to be generic and not specific to Alpine)?

Happy Easter!

Thanks, Trishank

On Fri, Mar 30, 2018 at 6:46 AM, Nick Coghlan wrote:

...

PEP link: https://www.python.org/dev/peps/pep-0571/

Thomas Kluyver has amended Mark Williams's PEP 571 to address the concerns & questions raised in the previous thread:

* manylinux2 -> manylinux2010:

https://github.com/python/peps/commit/70cbfda06534aedd6372f489090fdc8e1062de... * using glibc 2.12 as a compatibility marker:

https://github.com/python/peps/commit/d43b984e021eddc11bdbc36863c5c285b473f8...

(We also dropped a potentially misleading aside that could be taken as implying inaccurate limitations on Anaconda platform compatibility)

As I expect quite a few folks will be busy for the Easter weekend, I'm planning to accept the PEP a week from next Monday (i.e on April 9th) if no new concerns or objections are raised between now and then :)

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On 1 April 2018 at 01:18, Brett Cannon wrote:

On Fri, Mar 30, 2018, 09:13 Trishank Kuppusamy, < trishank.kuppusamy@datadoghq.com> wrote:

...

Hi Nick,

Long time no see --- hope everything is well on your side :)

First of all, thanks Mark, Thomas, and everyone else for your hard work on manylinux2010 --- I'm excited about this release, as I'm sure many others are!

This may be the wrong thread to discuss this,

It is, so I've forked the conversation 😉

and

...

I haven't looked too deeply into it, but are there plans to support container-specific Linux distros like Alpine that use musl instead of glibc?

Are there other container-specific distros other than Alpine that people use widely?

I'm guessing the answer is "no," but I'm curious to hear what others think

...

on the subject.

The discussion of distro-specific wheels has been brought up. I believe they are technically supported if you have your own index server, but PyPI doesn't, I believe, to keep an explosion of wheels people can't widely use down.

The main reason PyPI doesn't currently support distro specific wheels is because there isn't a compatibility tagging spec for them that's reasonable to use on a public index server - they currently end up being tagged as generic "linux" wheels. You can live with that on a private index server like the one that https://galaxyproject.org/ runs, but it would be incredibly confusing on PyPI. As far as I know, https://mail.python.org/pipermail/distutils-sig/2015-July/026617.html is still the most complete write-up we have of a potential way to fix that, which is to: 1. extend the default Linux platform tag to include the ID and VERSION_ID fields from /etc/os-release (credit to Galaxy Project's Nate Coraor for that core concept) 2. define a config file under /etc/python/ that distros can use to change both the build tag (e.g. allowing CentOS users to emit RHEL compatible wheels, or Ubuntu users to emit Debian compatible ones), and a list of binary compatible distro tags (e.g. allowing CentOS to consume RHEL tagged wheels, Ubuntu to consume Debian tagged wheels, and other musl based distros to consume Alpine tagged wheels) 3. define an equivalent per-virtualenv file to override the settings from (2) 4. update pip/setuptools/twine/warehouse/etc to account for the new compatibility tag variant One useful aspect here is that older clients will just ignored the new tags as "not a match", which means there shouldn't be any significant backwards compatibility hurdles.

Currently, if you want to cover all formats PyPI supports you need to produce 5 wheels. Adding a musl-based wheel format would bump that to 7 (as I assume musl supports 23 and 64-bit OSs).

For me, the questions of whether to support musl-based wheels on PyPI is will there be enough users to justify the cost to all project maintainers to produce 2 more wheels per release and what would the manylinux-equivalent spec look like (since it would have to be generic and not specific to Alpine)?

Honestly, I think the main benefit of heading down this road will be to allow folks to cache their custom wheel builds more effectively, so I'm not especially worried about making it generic (beyond the platform level config file suggested above). That said, if there were to be a significant growth in non-manylinux Linux wheels on PyPI, I'd expect them to be for the official Docker Inc base Python images, which are Alpine based, and hence can't use the glibc-based manylinux binaries. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 2 April 2018 at 17:45, Trishank Kuppusamy wrote:

...

That said, if there were to be a significant growth in non-manylinux Linux wheels on PyPI, I'd expect them to be for the official Docker Inc base Python images, which are Alpine based, and hence can't use the glibc-based manylinux binaries.

I think this is a good idea, especially given that manylinux1 and manylinux2010 wheels won't work on the official Docker image for Python. What does everyone else think?

My (mostly uninformed...) opinion is that having a mechanism to publish wheels for alternative compatibility classes (I hesitate to say "user defined" or "adhoc", because it implies too little structure, but I'm thinking in terms of something similar to manylinux, but without the need for a formal standards process) would be useful, because it would allow people to handle emerging standards like Alpine Linux for Docker, without going through a standardisation process, a pip/pypi release cycle, etc. I've no feel for how likely Alpine Linux is to still be as important as it currently is 12 months from now (maybe Docker will switch to an alternative base, who knows?) but we should be giving the community a means to react to such trends in a timely manner. Paul

Hello Nick, On Tue, Apr 3, 2018 at 7:59 AM, Nick Coghlan wrote:

At the specification level, we'll need an update to PEP 425 that's comparable to Dustin's PEP 566 for the core metadata (which both defined metadata 2.1, and made https://packaging.python.org/specifications/core-metadata/ the stable reference URL for the latest version of the metadata spec).

I'd like to see us do the same thing for platform compatibility tags: have https://packaging.python.org/specifications/platform- compatibility-tags/ define the format, and use the PEP process to make changes to that living spec.

While the primary motivating use case is distro-specific Linux wheels, we should at least consider the possibility of defining the new optional fields in compatibility tags in such a way that they could also be used for ABI compatibility levels in more centrally controlled platforms (i.e. Windows, macOS, iOS, Android), or to tag an expected deployment environment (e.g. letting the Galaxy Project folks mark all their internal wheels as "galaxyproject", and then configure their deployment environments to only accept wheels tagged that way).

Such a spec update would cover points 1->3 above.

At the implementation level, I don't have any great insight on how you might best go about pursuing that, but I suspect factoring out some useful helper functions into the low level `packaging` library may make sense (I believe Vinay Sajip's `distlib` already has some helpers along these lines that would potentially require updates). At a minimum, to be useful:

* either setuptools+bdist_wheel need to be able to produce wheels with the more detailed tags, or else there has be a suitable wheel renaming utility available (akin to `auditwheel repair`) * twine needs to be able to upload wheels with the more detailed tags * warehouse needs to be able to accept uploads with the more detailed tags

(For that last point, I'd personally prefer it to be a project level setting to opt-in to allowing uploads with the more detailed tags, such that the default behaviour is to reject them, and provide an opportunity for us to nudge folks towards manylinux wheels)

Sounds good. I'll take a stab at updating the PEP, and see. Can't make any promises as to a timeline, but I'll get back when (and if) I have a draft! Regards, Trishank

On 12 April 2018 at 21:54, Nick Coghlan wrote:

On 30 March 2018 at 20:46, Nick Coghlan wrote:

...

PEP link: https://www.python.org/dev/peps/pep-0571/

Thomas Kluyver has amended Mark Williams's PEP 571 to address the concerns & questions raised in the previous thread:

* manylinux2 -> manylinux2010: https://github.com/python/peps/commit/70cbfda06534aedd6372f489090fdc8e1062de... * using glibc 2.12 as a compatibility marker: https://github.com/python/peps/commit/d43b984e021eddc11bdbc36863c5c285b473f8...

(We also dropped a potentially misleading aside that could be taken as implying inaccurate limitations on Anaconda platform compatibility)

As I expect quite a few folks will be busy for the Easter weekend, I'm planning to accept the PEP a week from next Monday (i.e on April 9th) if no new concerns or objections are raised between now and then :)

With one small amendment [1] to appropriately credit Geoffrey Thomas and to provide a reference link for CalVer, I'm happy to report that I'm accepting the manylinux2010 specification :)

With the baseline ABI definition step out of the way, there's now the non-trivial task of getting this baseline to the point where folks can actually use it in practice to distribute pre-built binaries for their projects: https://github.com/pypa/manylinux/issues/179 If you think I've missed any essential steps from that tracking issue, or if there are other projects you think would be worth mentioning in the "Other projects to consider..." section, please let me know, either here, or on the issue. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia