[Distutils] Making pip and PyPI work with conda packages

[Paul Moore]

On 19 May 2015 at 00:25, Chris Barker <chris.barker at noaa.gov> wrote:
> Pretty much, yes. conda provides a way to package up and manage arbitrary
> stuff -- in this case, that would be non-python dependencies -- i.e. shared
> libs.
>
> So you can say that my_python_package depends on this_c_lib, and as long as
> you, or someone else has made a conda package for this_c_lib, then all is
> well.
>
> But python, setuptools, pip, wheel, etc. don't have a way to handle that
> shared lib as a dependency -- no standard way where to put it, no way to
> package it as a wheel, etc.
>
> So the way to deal with this with wheels is to statically link everything.
> But that's not how conda pa cakges are built, so no way to leverage conda
> here.

Thanks for the explanation. So, in effect, conda-as-a-platform defines
a (somewhat) incompatible platform for running Python, which can use
wheels just as python.org Python can, but which uses
conda-as-an-installer as its package manager (much like RPM or apt on
Unix).

The downside of this is that wheels built for conda (assuming that
it's OK to link with shared libs) are not compatible with python.org
builds (as those shared libs aren't available) and that difference
isn't reflected in the wheel ABI tags (and it's not particularly
clearly understood by the community, it seems). So publishing
conda-based wheels on PyPI would be a bad idea, because they wouldn't
work with python.org python (more precisely, only things that depend
on shared libs are affected, but the point remains).

> We need to remember what leveraging conda would buy us:
>
> conda doesn't actually make it any easier to build anything -- you need a
> platform-specific build script to build a conda package.
>
> conda does provide a way to manage non-python dependencies -- but that
> doesn't buy you anything unless you are using conda to manage your system
> anyway.
>
> conda DOES provide a community of people figuring out how to build complex
> packages, and building them, and putting them up for public dissemination.
>
> So the thing that leveraging conda can do is reduce the need for a lot of
> duplicated effort. And that effort is almost entirely about those third part
> libs -- after all, a compiled extension that has no dependencies is easy to
> build and put on PyPi. (OK, there is still a bit of duplicated effort in
> making the builds themselves on multiple platforms -- but with CI systems,
> that's not huge)

Agreed - that benefit would be significant (and not just in terms of
Python packaging - I'd personally find it useful in a lot of
non-Python projects), but it does rely on the information/scripts
being accessible in non-conda contexts. For example, for python.org
wheels, some way of modifying the build to create static libraries
rather than shared ones. I'm not sure to what extent the conda folks
would want to deal with handling the issues outside their own
environment, though (after all, working within a closed environment is
part of what makes the problem tractable for them).

The whole area of collecting build processes for common libraries is
an area that's always been badly managed on Windows, probably because
no-one has ever looked at it in a wider context than their own
project, and also because every library has its own "building on
Windows" solution (configure, custom MSVC solution files, CMake, etc).
And also because C/C++ has no concept anything like PyPI. But while
this is hugely interesting to me, it's not clear how well it fits with
Python packaging and distutils-sig (except as an unresolved C/C++
issue that we have to contend with).

Paul