[Distutils] Current Python packaging status (from my point of view)

Nathaniel Smith njs at pobox.com
Thu Nov 3 05:10:18 EDT 2016

On Nov 3, 2016 1:40 AM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
> On 3 November 2016 at 05:28, Nathaniel Smith <njs at pobox.com> wrote:
> > On Nov 2, 2016 9:52 AM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
> >> Tennessee Leeuwenberg started a draft PEP for that first part last
> >> year: https://github.com/pypa/interoperability-peps/pull/30/files
> >>
> >> dnf/yum, apt, brew, conda, et al all *work around* the current lack of
> >> such a system by asking humans (aka "downstream package maintainers")
> >> to supply the additional information by hand in a platform specific
> >> format.
> >
> > To be fair, though, it's not yet clear whether such a system is actually
> > possible. AFAIK no one has ever managed to reliably translate between
> > different languages that Linux distros use for describing environment
> > constraints, never mind handling the places where they're genuinely
> > irreconcilable (e.g. the way different distro openssl packages have
> > incompatible ABIs), plus other operating systems too.
> The main problem with mapping between Debian/RPM/conda etc in the
> general case is that the dependencies are generally expressed in terms
> of *names* rather than runtime actions, and you also encounter
> problems with different dependency management ecosystems splitting up
> (or aggregating!) different upstream components differently. This
> means you end up with a situation that's more like a lossy transcoding
> between MP3 and OGG Vorbis or vice-versa than it is a pristine
> encoding to either from a losslessly encoded FLAC or WAV file.
> The approach Tennessee and Robert Collins came up with (which still
> sounds sensible to me) is that instead of dependencies on particular
> external components, what we want to be able express is instead a
> range of *actions* that the software is going to take:
> - "I am going to dynamically load a library named <X>"
> - "I am going to execute a subprocess for a command named <Y>"

And then it segfaults because it turns out that your library named <X> is
not abi compatible with my library named <X>. Or it would have been if you
had the right version, but distros don't agree on how to express version
numbers either. (Just think about epochs.) Or we're on Windows, so it's
interesting to know that we need a library named <X>, but what are we
supposed to do with that information exactly?

Again, I don't want to just be throwing around stop energy -- if people
want to tackle these problems then I wish them luck. But I don't think we
should be hand waving this as a basically solved problem that just needs a
bit of coding, because that also can create stop energy that blocks an
honest evaluation of alternative approaches.

> And rather than expecting folks to figure that stuff out for
> themselves, you'd use tools like auditwheel and strace to find ways to
> generate it and/or validate it.
> > I mean, it would be awesome if someone pulls it off. But it's possible
> > this is like saying that it's not an inherent design limitation of my
> > that it's only suited for terrestrial use, because I could always strap
> > wings and a rocket...
> When it comes to things that stand in the way of fully automating the
> PyPI -> RPM pipeline, there are still a lot of *much* lower hanging
> fruit out there than this. However, the immediate incentives of folks
> working on package management line up in such a way that I'm
> reasonably confident that the lack of these capabilities is a matter
> of economic incentives rather than insurmountable technical barriers -
> it's a hard, tedious problem to automate, and manual repackaging into
> a platform specific format has long been a pretty easy thing for
> commercial open source redistributors to sell.

The idea does seem much more plausible as a form of metadata hints attached
to sdists that are used as inputs to a semi-automated-but-human-supervised
distro package building system. What I'm skeptical about is specifically
the idea that someday I'll be able to build a wheel, distribute it to end
users, and then pip will do some negotiation with
dnf/apt/pacman/chocolatey/whatever and make my wheel work everywhere -- and
that this will be an viable alternative to conda.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20161103/3651634a/attachment.html>

More information about the Distutils-SIG mailing list