[Distutils] Current Python packaging status (from my point of view)

[Nick Coghlan]

On 4 November 2016 at 07:44, Chris Barker <chris.barker at noaa.gov> wrote:
> On Thu, Nov 3, 2016 at 3:39 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> I don't think there's much chance of any of this ever working on
>> Windows - conda will rule there, and rightly so. Mac OS X seems likely
>> to go the same way, although there's an outside chance brew may pick
>> up some of the otherwise Linux-only capabilities if they prove
>> successful.
>
>
> this is a really good point then -- we're building  "platform independent
> system that, oh actually is a "a few different Linux vendors" system. Which
> is a perfectly worthy goal, but it was not at all the goal of conda.

conda needs raw material from software publishers, and it gets most of
that as software published in language specific formats on PyPI and
CRAN (which it then redistributes in both source and binary form),
*not* through original software being published specifically as conda
packages. That latter does happen, but it's the exception rather than
the rule.

That's *exactly* the same redistribution model that Linux distros use,
hence my semi-regular comparison between "I only want to publish and
support conda packages" and "We only want to support
CentOS/RHEL/Debian stable/Ubuntu LTS/insert your distro of choice".

Constraining the platforms we will offer free support for is a 100%
reasonable thing for volunteers to do, so it's entirely OK to tell
folks that use another platform "Sorry, I can't reproduce that
behaviour on a platform I support, so you're going to have to ask for
help in a platform specific forum", and then point them at a community
support forum like askubuntu.com or ask.fedoraproject.org, or else
nudge them in the direction of their vendor (if they're running a
commercially supported Python distribution). I do that myself - as a
concrete example, I knew the interaction between the -m switch and
multiprocessing on Windows was broken for years before I finally fixed
it when support for the multiprocessing "spawn" mode behaviour that is
used on Windows was added to the CPython Linux builds in Python 3.4.

>From a conda perspective though, the difference between Linux and
other deployment targets in the current situation is that one of the
competing CPython ABI providers is the operating system itself, while
on Windows it's mainly other commercial Python redistributors
(ActiveState, Enthought), and on Mac OS X it's mainly a community
driven package management community (brew) and a community run Python
redistributor (pyenv).

You're also up against the fact that most folks using Linux on a
client device are doing so by specific choice, and that Linux has been
the default supported system for open source projects for 20+ years.
This means that, despite what I said above, it's much easier in
practice for Python project participants to say they don't support and
can't help with PyPM/Enthought/brew/pyenv than it is to say that they
can't help folks running their software directly in the Linux system
Python.

Putting my work hat back on for a moment, I actually wish more people
*would* start saying that, as Red Hat actively want people to stop
running their own applications in the system Python, and start using
Software Collections (either directly or via the Docker images)
instead. Sharing a language runtime environment between operating
system components and end user applications creates all sorts of
maintainability problems (not least of which is the inability to
upgrade to new feature releases for fear of breaking end user
applications and vice-versa), to the point where Fedora is planning to
cede the "/usr/bin/python3" namespace to end users, and start
migrating system components out to "/usr/bin/system-python":
https://fedoraproject.org/wiki/Changes/System_Python

> I wasn't there, but I think conda started with the goal of supporting three
> core platforms (that are pretty different) with the same UI, and as much of
> the same code as possible. And if you want to do that -- it really makes a
> lot of sense to control as much of the system as possible.

Yep, hence my use of the phrase "cross-platform platform" - conda
defines a cross-platform API and build infrastructure for a few
different ABIs (e.g. x86_64 Linux, x32/x86_64 Windows, Mac OS X)

As a bit of additional background on this topic, folks that haven't
seen it may want to take a look at my "Python beyond CPython:
Adventures in Software Distribution" keynote from SciPy 2014,
especially the part starting from around 7:10 about software
redistribution networks: https://youtu.be/IVzjVqr_Bzs?t=458

The written speaker notes for that are available at
https://bitbucket.org/ncoghlan/misc/src/default/talks/2014-07-scipy/talk.md

>> It's not pointless, just hard,
>
> That I do see -- I see it as hard, so hard that it's likely to hit road
> blocks, so we could make better progress with other solutions if that's
> where people put their energies. And you said yourself earlier in this post
> that:
>
>  "I don't think there's much chance of any of this ever working on Windows"
>
> One of the "points" for me it to support Windows, OS-X and Linux -- so in
> that sense -- I guess it is pointless ;-)

For Windows and Mac OS X, my typical advice is "just use conda" and
pointing folks towards the Software Carpentry installation guide :)

The only time I deviate from that advice is if I know someone is
specifically interested in getting into professional Python
development as a paid gig, as in that context you're likely to need to
learn to be flexible about the base runtimes you work with.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia