[Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files

[Antoine Pitrou]

On Sun, 30 Nov 2014 01:47:16 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 29 November 2014 at 01:51, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Sat, 29 Nov 2014 01:27:44 +1000
> > Nick Coghlan <ncoghlan at gmail.com> wrote:
> >> >
> >> > Is this not going to be a slippery slope?
> >>
> >> Only if folks publish Linux binaries themselves, and that's still a
> >> bad idea (for the same reason publishing distro binaries is already a
> >> rare thing for people to do).
> >
> > Well, let's not make this a matter of ideology. Everyone knows it's a
> > bad idea to publish binaries, yet it's often better than nothing,
> > especially if the software is tedious to compile.
> 
> It's not a matter of ideology, but a matter of practicality. Debian
> stable, RHEL/CentOS, Ubuntu LTS, SLES - distros like these move slow
> enough (and have strong enough ABI compatibility guarantees) to be
> practical for ISVs to target with prebuilt binaries.

It seems we disagree on the notion of "practicality" :-)

For me practicality means being able to build a single binary package
for all recent Linux distros in a best effort approach. Building a
different package for each distro version is far from practical for any
reasonable-sized project (i.e. not something sponsored by a
1000+-employee entity, with a dedicated build team and infrastructure).

> > Case in point: can I ask you (the mythical "we") to build packages for
> > all major distros (including supported LTS releases), and the four most
> > recent Python versions, of the following piece of software:
> > https://github.com/numba/llvmlite
> 
> No, that would be a service provided by the as yet hypothetical PyPI
> build farm. If/when that happens, it will need to have a way of
> tagging Linux wheels appropriately, though.

"If/when that happens" is not reassuring, especially in the light of
how many pie-in-the-sky improvements in the packaging ecosystems have
turned out :-/

(at Continuum we have started offering such a service, but it's
"generic Linux": http://docs.binstar.org/build-config.html#BuildMatrix)

> Nearer term (and what prompted me to start this thread), the Fedora
> Environments & Stacks working group is investigating providing
> prebuilt wheel files for the Fedora ecosystem, and potentially for
> EPEL as well (see
> https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement
> for the broader context of that effort). For other ecosystems, you'll
> have to ask participants in those ecosystems.

That's asking software authors to complicate and slow down their
development process a lot. Also, there's no guarantee that Fedora or
Ubuntu or whatever would actually *accept* to help us, right?

> > I'm not sure I understand: distros provide their own packages, they
> > don't (shouldn't) blindly pull binary wheels from PyPI. Why would they
> > depend on the wheel tagging format?
> 
> We don't plan to blindly pull anything from PyPI - we're looking at
> the feasibility of publishing already reviewed software in ecosystem
> native formats (with the two pilot projects focusing on Java JAR files
> and Python wheel files).
> 
> When I last mentioned that idea here, Marcus pointed out that doing
> that with the generic "linux_x86_64" compatibility tag on the wheel
> filenames would be problematic, as there'd be nothing preventing
> anyone from pulling them down onto inappropriate systems, with no
> obvious trace of the Fedora or EPEL specific assumptions that went
> into building them.

Uh, there's a lot of hidden knowledge required to understand those
two paragraphs that I don't master. I don't know what "inappropriate
systems" are, what are "reviewed software", etc. ;-)

Also I don't understand why you're not recompiling as you would
normally do.

> While that's a valid concern, I also don't want to go invent our own
> custom compatibility tagging convention just for Fedora & EPEL, but
> rather work within the limits of what upstream Python packaging
> natively supports.

Well, *allowing* distro tags in the platform tag is certainly ok. What
I'm afraid of is if that's made mandatory.

> Consider if the following could be included in the "pyvenv.cfg" file
> in a virtual environment:
> 
> [compatibility]
> python=cp27,cp2,py2
> abi=cp27mu
> platform=linux_x86_64_epel_6
> 
> Or for a Python 3 virtual environment:
> 
> [compatibility]
> python=cp34,cp3,py3
> abi=cp34m,abi3
> platform=linux_x86_64_epel_6
> 
> If present, these pyvenv.cfg settings would override the normal PEP
> 425 compatibility tag calculations (I'm OK with the idea of *needing*
> to be in a virtual environment to gain the power to configure these
> tags).

As long as it's on an opt-in basis, it certainly sounds ok.

Regards

Antoine.