[Distutils] Any thoughts on supporting e.g. Alpine Linux? (was: PEP 571: Any further manylinux2010 concerns or questions?

[Nick Coghlan]

On 3 April 2018 at 02:45, Trishank Kuppusamy
<trishank.kuppusamy at datadoghq.com> wrote:
> On Sat, Mar 31, 2018 at 11:06 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> The main reason PyPI doesn't currently support distro specific wheels is
>> because there isn't a compatibility tagging spec for them that's reasonable
>> to use on a public index server - they currently end up being tagged as
>> generic "linux" wheels. You can live with that on a private index server
>> like the one that https://galaxyproject.org/ runs, but it would be
>> incredibly confusing on PyPI.
>>
>> As far as I know,
>> https://mail.python.org/pipermail/distutils-sig/2015-July/026617.html is
>> still the most complete write-up we have of a potential way to fix that,
>> which is to:
>>
>> 1. extend the default Linux platform tag to include the ID and VERSION_ID
>> fields from /etc/os-release (credit to Galaxy Project's Nate Coraor for that
>> core concept)
>> 2. define a config file under /etc/python/ that distros can use to change
>> both the build tag (e.g. allowing CentOS users to emit RHEL compatible
>> wheels, or Ubuntu users to emit Debian compatible ones), and a list of
>> binary compatible distro tags (e.g. allowing CentOS to consume RHEL tagged
>> wheels, Ubuntu to consume Debian tagged wheels, and other musl based distros
>> to consume Alpine tagged wheels)
>> 3. define an equivalent per-virtualenv file to override the settings from
>> (2)
>> 4. update pip/setuptools/twine/warehouse/etc to account for the new
>> compatibility tag variant

[snip]

> If we wanted to contribute towards this effort, how can the community help,
> Nick?

At the specification level, we'll need an update to PEP 425 that's
comparable to Dustin's PEP 566 for the core metadata (which both
defined metadata 2.1, and made
https://packaging.python.org/specifications/core-metadata/ the stable
reference URL for the latest version of the metadata spec).

I'd like to see us do the same thing for platform compatibility tags:
have https://packaging.python.org/specifications/platform-compatibility-tags/
define the format, and use the PEP process to make changes to that
living spec.

While the primary motivating use case is distro-specific Linux wheels,
we should at least consider the possibility of defining the new
optional fields in compatibility tags in such a way that they could
also be used for ABI compatibility levels in more centrally controlled
platforms (i.e. Windows, macOS, iOS, Android), or to tag an expected
deployment environment (e.g. letting the Galaxy Project folks mark all
their internal wheels as "galaxyproject", and then configure their
deployment environments to only accept wheels tagged that way).

Such a spec update would cover points 1->3 above.

At the implementation level, I don't have any great insight on how you
might best go about pursuing that, but I suspect factoring out some
useful helper functions into the low level `packaging` library may
make sense (I believe Vinay Sajip's `distlib` already has some helpers
along these lines that would potentially require updates). At a
minimum, to be useful:

* either setuptools+bdist_wheel need to be able to produce wheels with
the more detailed tags, or else there has be a suitable wheel renaming
utility available (akin to `auditwheel repair`)
* twine needs to be able to upload wheels with the more detailed tags
* warehouse needs to be able to accept uploads with the more detailed tags

(For that last point, I'd personally prefer it to be a project level
setting to opt-in to allowing uploads with the more detailed tags,
such that the default behaviour is to reject them, and provide an
opportunity for us to nudge folks towards manylinux wheels)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia