[Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files

[Nick Coghlan]

On 30 November 2014 at 02:10, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sun, 30 Nov 2014 01:47:16 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 29 November 2014 at 01:51, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> > On Sat, 29 Nov 2014 01:27:44 +1000
>> > Nick Coghlan <ncoghlan at gmail.com> wrote:
>> >> >
>> >> > Is this not going to be a slippery slope?
>> >>
>> >> Only if folks publish Linux binaries themselves, and that's still a
>> >> bad idea (for the same reason publishing distro binaries is already a
>> >> rare thing for people to do).
>> >
>> > Well, let's not make this a matter of ideology. Everyone knows it's a
>> > bad idea to publish binaries, yet it's often better than nothing,
>> > especially if the software is tedious to compile.
>>
>> It's not a matter of ideology, but a matter of practicality. Debian
>> stable, RHEL/CentOS, Ubuntu LTS, SLES - distros like these move slow
>> enough (and have strong enough ABI compatibility guarantees) to be
>> practical for ISVs to target with prebuilt binaries.
>
> It seems we disagree on the notion of "practicality" :-)

Note I said "ISVs" there - folks that actually make money from
targetting those platforms. I don't think providing integrated Linux
binaries is currently practical at all for open source projects (note
that even CPython just provides a tarball upstream, with the builds
handled directly by the distros).

> For me practicality means being able to build a single binary package
> for all recent Linux distros in a best effort approach.

That isn't realistically possible for anything that isn't completely
statically linked, as distro level choices of build options vary too
much.

We could technically allow statically linked binaries on PyPI by
dropping the restriction against generic Linux wheel files, but I'm
wary of that in the absence of an automated server side scan for
dynamically linked binaries.

Many users (quite reasonably, if they're primarily Python developers)
have problems working through build failures when attempting to
install non-Python extensions from source. Such build failures are
usually models of clarity compared to diagnosing dynamic linking
failures.

> Building a
> different package for each distro version is far from practical for any
> reasonable-sized project (i.e. not something sponsored by a
> 1000+-employee entity, with a dedicated build team and infrastructure).

This is why I'm saying people *shouldn't* try to provide prebuilt
Linux binaries in the general case: it's simply not practical. Hence
the rise of higher level, more self-contained formats like Docker, as
well as the fact that I'm not even trying to solve the "any Linux"
problem myself, but going after the vastly simpler problem of
targeting the ecosystem I need to target (but hopefully in a
sufficiently general purpose way that other ecosystems can adopt the
same model if they so choose).

>> > Case in point: can I ask you (the mythical "we") to build packages for
>> > all major distros (including supported LTS releases), and the four most
>> > recent Python versions, of the following piece of software:
>> > https://github.com/numba/llvmlite
>>
>> No, that would be a service provided by the as yet hypothetical PyPI
>> build farm. If/when that happens, it will need to have a way of
>> tagging Linux wheels appropriately, though.
>
> "If/when that happens" is not reassuring, especially in the light of
> how many pie-in-the-sky improvements in the packaging ecosystems have
> turned out :-/

Nobody is currently offering to provide paid staff to work on it, so
it won't happen until it gets to the top of volunteers' personal
priority lists.

> (at Continuum we have started offering such a service, but it's
> "generic Linux": http://docs.binstar.org/build-config.html#BuildMatrix)

Yes, Continuum avoided the distro ABI compatibility problem by
defining its own ABI. It's exactly the same model I'm proposing - in
the general case, you can't take Continuum built packages, and use
them with Fedora/Debian/etc built dynamically linked dependencies.
Same with my suggestion - it's about labelling the compatibility
requirements, rather than trying to avoid them by statically linking
everything.

>> Nearer term (and what prompted me to start this thread), the Fedora
>> Environments & Stacks working group is investigating providing
>> prebuilt wheel files for the Fedora ecosystem, and potentially for
>> EPEL as well (see
>> https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement
>> for the broader context of that effort). For other ecosystems, you'll
>> have to ask participants in those ecosystems.
>
> That's asking software authors to complicate and slow down their
> development process a lot.

If the distro level service doesn't benefit them, there's no reason
for software authors to use it. This is about adding a new option for
consuming software, not taking anything away.

In my case, I won't deploy something until it's been through at least
a basic licensing review, and if we're doing that work anyway, I may
as well see if we can find a way to do it upstream in Fedora rather
than inside the Red Hat firewall. Same for prebuilding binary wheels -
sure, we could do that just for our own internal infrastructure, but
it makes more sense to me to at least investigate the idea of handling
it upstream first.

> Also, there's no guarantee that Fedora or
> Ubuntu or whatever would actually *accept* to help us, right?

For Fedora, myself and Slavek (the Fedora Python maintainer) are the
two folks working on this (& we're both voting members of the
Environments & Stacks working group). So yes, in Fedora's case, it's
definitely a developer experience problem we want to solve properly,
and we're already working on it - this is me asking upstream for help
resolving a design problem, rather than coming up with a speculative
idea to persuade Fedora to adopt later.

I realise that's very different from the historical attitude of Linux
distributions to upstream packaging ecosystems. Getting to this point
of adopting a more user-focused mindset when asking what services a
Linux distro should be offering has been a formidable political
challenge involving a lot of work from a lot of different people
(including the current Fedora Project Leader), so I'm not even going
to speculate on what might be involved in attempting to replicate that
in the context of a different distro ecosystem :)

>> > I'm not sure I understand: distros provide their own packages, they
>> > don't (shouldn't) blindly pull binary wheels from PyPI. Why would they
>> > depend on the wheel tagging format?
>>
>> We don't plan to blindly pull anything from PyPI - we're looking at
>> the feasibility of publishing already reviewed software in ecosystem
>> native formats (with the two pilot projects focusing on Java JAR files
>> and Python wheel files).
>>
>> When I last mentioned that idea here, Marcus pointed out that doing
>> that with the generic "linux_x86_64" compatibility tag on the wheel
>> filenames would be problematic, as there'd be nothing preventing
>> anyone from pulling them down onto inappropriate systems, with no
>> obvious trace of the Fedora or EPEL specific assumptions that went
>> into building them.
>
> Uh, there's a lot of hidden knowledge required to understand those
> two paragraphs that I don't master. I don't know what "inappropriate
> systems" are, what are "reviewed software", etc. ;-)

An example of inappropriate systems: trying to install a Fedora wheel
on Ubuntu. That often won't work in the general case, but pip will
currently allow it (because they share a platform tag, despite not
sharing a platform ABI)

"Reviewed software" just refers to the Fedora package set. Currently
we only make that available for installation as RPMs, the Environments
& Stacks WG is looking at making it available in more developer
friendly formats, rather than focusing solely on the operations use
case. (We're not sure yet how we'll make that sustainable from a build
infrastructure perspective, but that's a large part of what the pilot
projects are aiming to figure out)

> Also I don't understand why you're not recompiling as you would
> normally do.

We'll still be recompiling. We just want to publish the results for
Python packages as virtual-environment compatible wheel files, not
just as RPMs.

>> While that's a valid concern, I also don't want to go invent our own
>> custom compatibility tagging convention just for Fedora & EPEL, but
>> rather work within the limits of what upstream Python packaging
>> natively supports.
>
> Well, *allowing* distro tags in the platform tag is certainly ok. What
> I'm afraid of is if that's made mandatory.

OK, that makes more sense. Yes, I agree we need to keep the ability to
say "this is a prebuilt, self-contained, binary wheel that should run
on any Linux system because it doesn't link to any system binaries".
Chalk it up as yet another reason that the specific proposal I started
the thread with wouldn't actually work :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia