[Distutils] Current Python packaging status (from my point of view)

Nick Coghlan ncoghlan at gmail.com
Thu Nov 3 06:39:32 EDT 2016


On 3 November 2016 at 19:10, Nathaniel Smith <njs at pobox.com> wrote:
> On Nov 3, 2016 1:40 AM, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>> The approach Tennessee and Robert Collins came up with (which still
>> sounds sensible to me) is that instead of dependencies on particular
>> external components, what we want to be able express is instead a
>> range of *actions* that the software is going to take:
>>
>> - "I am going to dynamically load a library named <X>"
>> - "I am going to execute a subprocess for a command named <Y>"
>
> And then it segfaults because it turns out that your library named <X> is
> not abi compatible with my library named <X>. Or it would have been if you
> had the right version, but distros don't agree on how to express version
> numbers either. (Just think about epochs.) Or we're on Windows, so it's
> interesting to know that we need a library named <X>, but what are we
> supposed to do with that information exactly?

I don't think there's much chance of any of this ever working on
Windows - conda will rule there, and rightly so. Mac OS X seems likely
to go the same way, although there's an outside chance brew may pick
up some of the otherwise Linux-only capabilities if they prove
successful.

Linux distros have larger package management communities than conda
though, and excellent integration testing infrastructure on the
commercial side of things, so we "just" need to get some of their
software integration pipelines arranged in better configurations, and
responding to the right software publication events :)

> Again, I don't want to just be throwing around stop energy -- if people want
> to tackle these problems then I wish them luck. But I don't think we should
> be hand waving this as a basically solved problem that just needs a bit of
> coding, because that also can create stop energy that blocks an honest
> evaluation of alternative approaches.

There's a reason I'm not volunteering to work on this in my own time,
and am instead letting the Fedora Modularity and Python maintenance
folks decide when it's a major blocker to further process improvement
efforts on their side of things, while I work on other
non-Python-specific aspects of Red Hat's software supply chain
management.

What I'm mainly reacting to here is Chris's apparent position that he
sees the entire effort as pointless. It's not pointless, just hard,
since it involves working towards process improvements that span
*multiple* open source communities, rather than being able to work
with just the Python community or just the conda community. Even if
automation can only cover 80-90% of legacy cases and 95% of future
cases (numbers plucked out of thin air), that's still a huge
improvement since folks generally aren't adding *new* huge-and-gnarly
C/C++/FORTRAN dependencies to the mix of things pip users need to
worry about, while Go and Rust have been paying much closer attention
to supporting distributed automated builds from the start of their
existence.

>> When it comes to things that stand in the way of fully automating the
>> PyPI -> RPM pipeline, there are still a lot of *much* lower hanging
>> fruit out there than this. However, the immediate incentives of folks
>> working on package management line up in such a way that I'm
>> reasonably confident that the lack of these capabilities is a matter
>> of economic incentives rather than insurmountable technical barriers -
>> it's a hard, tedious problem to automate, and manual repackaging into
>> a platform specific format has long been a pretty easy thing for
>> commercial open source redistributors to sell.
>
> The idea does seem much more plausible as a form of metadata hints attached
> to sdists that are used as inputs to a semi-automated-but-human-supervised
> distro package building system. What I'm skeptical about is specifically the
> idea that someday I'll be able to build a wheel, distribute it to end users,
> and then pip will do some negotiation with
> dnf/apt/pacman/chocolatey/whatever and make my wheel work everywhere -- and
> that this will be an viable alternative to conda.

Not in the general case it won't, no, but it should be entirely
feasible for distro-specific Linux wheels and manylinux1 wheels to be
as user-friendly as conda in cases where folks on the distro side are
willing to put in the effort to make the integration work properly.

It may also be feasible to define an ABI for "linuxconda" that's
broader than the manylinux1 ABI, so folks can publish conda wheels
direct to PyPI, and then pip could define a way for distros to
indicate their ABI is "conda compatible" somehow.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


More information about the Distutils-SIG mailing list