[Python-Dev] Software integrators vs end users (was Re: Language Summit notes)

[Nick Coghlan]

On 18 April 2014 15:39, Donald Stufft <donald at stufft.io> wrote:
>
> On Apr 18, 2014, at 3:18 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> At this point, however, I'm mainly looking for consensus that there
>> *are* two different problems to be solved here, and solving them both
>> well in a single tool is likely to be nigh on impossible. (I'm aware
>> we're really on the wrong list for that discussion, but I also think
>> there's value in getting some broader python-dev awareness of this
>> particular topic)
>
> I’m not sure about this? I mean yes those are two different areas, but I’m
> not sure about the split between Conda and pip here. As far as I can tell
> Conda is useful in the same way apt-get or yum is useful, you get a non
> Python specific set of packages (because sometimes things aren’t pure
> python) and you also remove a little bit of thinking about versions (although
> honestly I think it’s possible to remove most of that for consumers of
> packages).

You also get the ability to use that same system to update *Python
itself*, regardless of platform. Being notified of and consuming
CPython updates on Windows, as well as consuming alternate versions on
Linux distros, is currently not a well solved problem - with conda,
it's no more complex than updating a PyPI package.

That's one of the most attractive aspects for me - making Python 3.4,
pypy, etc as easy to update with consistent cross platform
instructions as Python packages are.

> To be quite frank, a lot of the benefit of Conda outside of the “I need something
> that isn’t strictly Python) is in the fact they can bootstrap compiled packages
> whereas pip/wheel/PyPI combination we need to convince authors to upload
> their own binary packages (or at least develop something to make it easier
> like a build farm).

Yep, that's a large part of why I think "divide and conquer" is the
way to go here. While it isn't completely accurate (as most SaaS
developers don't want to build C extensions from source) I think
Guido's "build from source" vs "install a pre-built binary"
distinction is still a reasonably good way to characterise it. For a
distro like Fedora (or, even more so, RHEL), we're not going to trust
a binary created by someone else if we can possibly avoid it, so
upstream binaries aren't useful to us, but pip's ability to abstract
away the vagaries of the upcoming metadata 2.0 migration is incredibly
helpful. The other distros are in the same situation (we'll always
feed source tarballs into our own build systems), and ditto for the
conda folks. We need that for all sorts of reasons that potential new
Python users don't care about, but continuing to meet our
requirements, along with the free-for-all that is PyPI makes handling
the binary distribution problem *much* harder for pip.

By contrast, like any other distro, conda doesn't need to boil the
ocean - it just needs to provide a filtered, up to date set of core
packages that work well together. The advantage it has over other
distros is that it is *cross-platform* - it works essentially the same
way on Windows, Mac OS X. Most other package management systems are
either platform specific or can't handle arbitrary binary
dependencies. By being Python-centric (even if not Python specific),
there's also a strong focus on updating the core packages more often
than the Linux distros do.

There's no "always use conda instead of pip" competition here - we
need pip, we need wheels. But I see those as software integrator
focused tools that would need a lot of additional work to provide a
truly spectacularly compelling out of the box experience for new
users. I don't think that's a useful way to expend effort - I think it
makes more sense to work on a separate "here's the fully assembled
environment for using Python as a tool to develop ideas" introduction,
while pushing the "here's how to build your own custom environment
from upstream parts" as a later step in a new user's journey towards
full Python mastery (if they're interested in that path). Making sure
that "pip install foo" does the right thing inside conda environments
(if it doesn't already) should be all that is needed to ensure that
random installation instructions on the internet still work.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia