[Distutils] Handling the binary dependency management problem

Nick Coghlan ncoghlan at gmail.com
Tue Dec 3 11:19:08 CET 2013

On 3 December 2013 19:22, Paul Moore <p.f.moore at gmail.com> wrote:
> On 3 December 2013 08:48, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> This means that one key reason I want to recommend it for the cases where it
>> is a good fit (i.e. the scientific Python stack) is so we can explicitly
>> advise *against* using it in other cases where it will just add complexity
>> without adding value.
>> Saying nothing is not an option, since people are already confused. Saying
>> to never use it isn't an option either, since bootstrapping conda first *is*
>> a substantially simpler cross-platform way to get up to date scientific
>> Python software on to your system. The alternatives are platform specific
>> and (at least in the Linux distro case) slower to get updates.
> But you're not saying "use conda for the scientific Python stack".
> You're saying to use it "when you have binary external dependencies"
> which is a phrase that I (and I suspect many Windows users) don't
> really understand and will take to mean "C extensions, or at least
> ones that interface to another library, sich as pyyaml, lxml, ...)

That's not what I meant though - I only mean the case where there's a
binary dependency that's completely outside the Python ecosystem and
can't be linked or bundled because it needs to be shared between
multiple components on the Python side.

However, there haven't been any compelling examples presented other
than the C runtime (which wheel needs to handle as part of the
platform tag and/or the ABI tag) and the scientific stack, so I agree
limiting the recommendation to the scientific stack is a reasonable
approach. Only folks that actually understand the difference between
static and dynamic linking and wrapper modules vs self-contained
accelerator modules are likely to understand what "shared external
binary dependency" means, so I agree it's not a useful phrase to use
in a recommendation aimed at folks that aren't already experienced

If Windows and Mac OS X users have alternatives they strongly favour
over conda that are virtualenv compatible, then sure, we can consider
those as well, but I'm not aware of any (as the "virtualenv
compatible" bit rules out anything based on platform installers).

> Also, this presumes an either/or situation. What about someone who
> just wants to use matplotlib to display a graph of some business data?
> Is matplotlib part of "the scientific stack"? Should I use conda
> *just* to get matplotlib in an otherwise wheel-based application?

Ultimately, it depends on if matplotlib is coupled to the NumPy build
options or not. However, I think the more practical recommendation
would be to say:

- if there's no wheel
- and you can't build it from source yourself
- then you can try "pip install conda && conda init && conda install
<pkg>" as a fallback option.

And then we encourage the conda devs to follow the installation
database standard properly (if they aren't already), so things
installed with conda play nice with things installed with pip.

It sounds like we also need to get them to ensure they're using the
right compiler/C runtime on Windows so their packages are
interoperable with the standard python.org installers.

> Or
> how about a scientist that wants wxPython (to use Chris' example)?
> Apparently the conda repo doesn't include wxPython, so do they need to
> learn how to install pip into a conda environment? (Note that there's
> no wxPython wheel, so this isn't a good example yet, but I'd hope it
> will be in due course...)

No, it's the other way around - for cases where wheels aren't yet
available, but conda provides it, then we should try to ensure that
"pip install conda && conda init && conda install <package>" does the
right thing (including conda upgrading previously pip installed
packages when necessary, as well as bailing out gracefully when it
needs to).

At the moment, we're getting people trying to use conda as the base,
and stuff falling apart at a later stage, since conda isn't structured
properly to handle use cases other than the scientific one where
simplicity and repeatabilitly for people that aren't primarily
developers trumps platform integration and easier handling of security

> Reducing confusion is good, I'm all for that. But we need to have a
> clear picture of what we're saying before we can state it clearly...

Agreed, that's a large part of why I started this thread. It's
definitely clarified several points for me.


Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

More information about the Distutils-SIG mailing list