[Distutils] Current Python packaging status (from my point of view)

[Nick Coghlan]

On 3 November 2016 at 05:28, Nathaniel Smith <njs at pobox.com> wrote:
> On Nov 2, 2016 9:52 AM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>> Tennessee Leeuwenberg started a draft PEP for that first part last
>> year: https://github.com/pypa/interoperability-peps/pull/30/files
>>
>> dnf/yum, apt, brew, conda, et al all *work around* the current lack of
>> such a system by asking humans (aka "downstream package maintainers")
>> to supply the additional information by hand in a platform specific
>> format.
>
> To be fair, though, it's not yet clear whether such a system is actually
> possible. AFAIK no one has ever managed to reliably translate between
> different languages that Linux distros use for describing environment
> constraints, never mind handling the places where they're genuinely
> irreconcilable (e.g. the way different distro openssl packages have
> incompatible ABIs), plus other operating systems too.

The main problem with mapping between Debian/RPM/conda etc in the
general case is that the dependencies are generally expressed in terms
of *names* rather than runtime actions, and you also encounter
problems with different dependency management ecosystems splitting up
(or aggregating!) different upstream components differently. This
means you end up with a situation that's more like a lossy transcoding
between MP3 and OGG Vorbis or vice-versa than it is a pristine
encoding to either from a losslessly encoded FLAC or WAV file.

The approach Tennessee and Robert Collins came up with (which still
sounds sensible to me) is that instead of dependencies on particular
external components, what we want to be able express is instead a
range of *actions* that the software is going to take:

- "I am going to dynamically load a library named <X>"
- "I am going to execute a subprocess for a command named <Y>"

And rather than expecting folks to figure that stuff out for
themselves, you'd use tools like auditwheel and strace to find ways to
generate it and/or validate it.

> I mean, it would be awesome if someone pulls it off. But it's possible that
> this is like saying that it's not an inherent design limitation of my bike
> that it's only suited for terrestrial use, because I could always strap on
> wings and a rocket...

When it comes to things that stand in the way of fully automating the
PyPI -> RPM pipeline, there are still a lot of *much* lower hanging
fruit out there than this. However, the immediate incentives of folks
working on package management line up in such a way that I'm
reasonably confident that the lack of these capabilities is a matter
of economic incentives rather than insurmountable technical barriers -
it's a hard, tedious problem to automate, and manual repackaging into
a platform specific format has long been a pretty easy thing for
commercial open source redistributors to sell.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia