[Distutils] Any thoughts on supporting e.g. Alpine Linux? (was: PEP 571: Any further manylinux2010 concerns or questions?

[Nick Coghlan]

On 1 April 2018 at 01:18, Brett Cannon <brett at python.org> wrote:

>
>
> On Fri, Mar 30, 2018, 09:13 Trishank Kuppusamy, <
> trishank.kuppusamy at datadoghq.com> wrote:
>
>> Hi Nick,
>>
>> Long time no see --- hope everything is well on your side :)
>>
>> First of all, thanks Mark, Thomas, and everyone else for your hard work
>> on manylinux2010 --- I'm excited about this release, as I'm sure many
>> others are!
>>
>> This may be the wrong thread to discuss this,
>>
>
> It is, so I've forked the conversation 😉
>
> and
>
>> I haven't looked too deeply into it, but are there plans to support
>> container-specific Linux distros like Alpine that use musl instead of glibc?
>>
>
> Are there other container-specific distros other than Alpine that people
> use widely?
>
> I'm guessing the answer is "no," but I'm curious to hear what others think
>> on the subject.
>>
>
> The discussion of distro-specific wheels has been brought up. I believe
> they are technically supported if you have your own index server, but PyPI
> doesn't, I believe, to keep an explosion of wheels people can't widely use
> down.
>

The main reason PyPI doesn't currently support distro specific wheels is
because there isn't a compatibility tagging spec for them that's reasonable
to use on a public index server - they currently end up being tagged as
generic "linux" wheels. You can live with that on a private index server
like the one that https://galaxyproject.org/ runs, but it would be
incredibly confusing on PyPI.

As far as I know,
https://mail.python.org/pipermail/distutils-sig/2015-July/026617.html is
still the most complete write-up we have of a potential way to fix that,
which is to:

1. extend the default Linux platform tag to include the ID and VERSION_ID
fields from /etc/os-release (credit to Galaxy Project's Nate Coraor for
that core concept)
2. define a config file under /etc/python/ that distros can use to change
both the build tag (e.g. allowing CentOS users to emit RHEL compatible
wheels, or Ubuntu users to emit Debian compatible ones), and a list of
binary compatible distro tags (e.g. allowing CentOS to consume RHEL tagged
wheels, Ubuntu to consume Debian tagged wheels, and other musl based
distros to consume Alpine tagged wheels)
3. define an equivalent per-virtualenv file to override the settings from
(2)
4. update pip/setuptools/twine/warehouse/etc to account for the new
compatibility tag variant

One useful aspect here is that older clients will just ignored the new tags
as "not a match", which means there shouldn't be any significant backwards
compatibility hurdles.


> Currently, if you want to cover all formats PyPI supports you need to
> produce 5 wheels. Adding a musl-based wheel format would bump that to 7 (as
> I assume musl supports 23 and 64-bit OSs).
>
> For me, the questions of whether to support musl-based wheels on PyPI is
> will there be enough users to justify the cost to all project maintainers
> to produce 2 more wheels per release and what would the
> manylinux-equivalent spec look like (since it would have to be generic and
> not specific to Alpine)?
>

Honestly, I think the main benefit of heading down this road will be to
allow folks to cache their custom wheel builds more effectively, so I'm not
especially worried about making it generic (beyond the platform level
config file suggested above).

That said, if there were to be a significant growth in non-manylinux Linux
wheels on PyPI, I'd expect them to be for the official Docker Inc base
Python images, which are Alpine based, and hence can't use the glibc-based
manylinux binaries.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20180401/417ebf8b/attachment.html>