[Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

[Nick Coghlan]

On 19 May 2015 at 09:43, Chris Barker <chris.barker at noaa.gov> wrote:
> On Mon, May 18, 2015 at 11:21 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>> My honest view is that unless conda is intending to replace pip and
>> wheel totally, you cannot assume that people will be happy to use
>> conda alongside pip (or indeed, use any pair of independent packaging
>> tools together - people typically want one unified solution). And if
>> the scientific community stops working towards providing wheels for
>> people without compilers "because you can use conda", there is going
>> to be a proportion of the Python community that will lose out on some
>> great tools as a result.
>
>
> Exactly -- this idea that there are two (or more) non-overlapping
> communities is pretty destructive.

There's a cornucopia of *overlapping* communities. We only rarely hear
from system administrators upstream, for example, as they tend to be
mainly invested in particular operating system or configuration
management communities, leaving upstream mostly to developers and data
analysts. For these admins, a package management system is only going
to be potentially interesting if it is supported by their operating
system or configuration management tool of choice (e.g.
http://docs.ansible.com/list_of_packaging_modules.html for Ansible, or
some of the options linked from Salt's package management abstraction
layer: http://docs.saltstack.com/en/latest/ref/states/all/salt.states.pkg.html)

This is why I'm such a big fan of richer upstream metadata with
automated conversion to downstream formats as my preferred long term
solution - this isn't a "pip vs conda" story, it's "pip vs conda vs
yum vs apt vs MSI vs nix vs zypper vs zc.buildout vs enstaller vs PyPM
vs ....". (in addition to the modules listed for Ansible and Salt, I
discovered yet another one today: https://labix.org/smart)

The main differences I see with conda relative to the other downstream
package management systems is that it happened to be made by folks
that are also heavily involved in development of Python based data
analysis tools, and that some of its proponents want it to be the "one
package management tool to rule them all". I consider the latter
proposal to be as outlandish an idea as believing the world only needs
one programming language - just as with programming languages,
packaging system design involves making trade-offs between different
priorities, so you can't optimise for everything at once. conda's an
excellent cross-platform end user focused dependency management
system. This is a good thing, but it does mean conda isn't a suitable
candidate for use as an input format for other tools that compete with
it.

As far as the "we could use a better dynamic linking story for Windows
and Mac OS X" story goes, now that I understand the general *nix case
is considered out of scope for the situations Chris is interested in,
I think there's a reasonable case to be made for being able to
*bundle* redistributable dynamically linked libraries with a wheel
file, and for the build process of *other* wheel files to be able to
rely on those bundled external libraries. I originally thought the
request was about being able to *describe* the external dependencies
in sufficient detail that the general case on *nix could be handled,
or that an appropriate Windows or Mac OS X binary could be obtained
out of band, rather than by being bundled with the relevant wheel
file.

Getting a bundling based model to work reliably is still going to be
difficult (and definitely more complicated than static linking in
cases where data sharing isn't needed), but it's not intractable the
way the general case is.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia