[Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

[Paul Moore]

On 21 May 2015 at 16:37, Donald Stufft <donald at stufft.io> wrote:
> Also, one of the goals a few of us has in the PyPA is that we move to a
> future where the build systems are  pluggable. So one package could be
> building using setuptools, another building using some SciPy specific build
> tool, another using a whole other one. They will all ideally have some sort
> of generic interface that they need to work with, but using pip means you
> get the details of abstracting out to the different build tools handled for
> you, for “free”. At least, in theory that’s how it’ll work :)

Note that this is a key to why wheel is important to this discussion.
The "build interface" in pip is "pip wheel foo", which will (in the
"pluggable build" future) run whatever build tool the project
specifies, and produce as output a wheel. That wheel is then the input
for any packaging systems that want to build their own formats.

So, we have:

1. sdist: The source format for packages.
2. distutils/setuptools/bdist_wheel: The only major "build tool"
currently in existence. Our goal is to seamlessly allow others to fit
here.
3. pip wheel: The build tool interface designed to convert
sdist->wheel using the appropriate build tool.
4. wheel: The built format for Python packages. Acts as a common
target for build tools and as a common source for distribution package
builders. Also directly installable via pip.
5. pip install <wheelfile>. The canonical Python installer, taking
wheels as input.

Pip can also combine this whole sequence, and install direct from
sdist (via wheel in the next version, currently by direct install from
sdist), but that;s not the important point for this discussion.

In terms of tool interoperability, therefore, there are a couple of
places things can hook in.

1. Anything that can take an sdist and build a wheel can be treated as
a "build tool". You could run the appropriate build tool for your
package manually, but it's a goal for pip to provide a unified
interface to that process.
2. Any installer can use wheels as the source for building its install
packages. This frees package build processes from needing to deal with
compiling Python extensions, packaging up Python sources, etc.

We (the PyPA) haven't really done a particularly good job of
articulating this design, not least because a lot of it is still ideas
in progress, rather than concrete plans. And as a result, it's hard
for tools like conda to clearly understand how they could fit into
this stack. And of course, the backward compatibility pressures on any
change in Python packaging causes things to go pretty slowly, meaning
that projects like conda have an additional pressure to come up with a
solution *right now* rather than waiting for a standard solution that
frankly is currently vapourware.

Ideally, the scientific community's experiences with building complex
Python packages can help us to improve the wheel spec to ensure that
it can better act as that universal binary format (for repackaging or
direct installation). But that does require ongoing effort to make
sure we understand where the wheel format falls short, and how we can
fix those issues. Doing this without getting sucked into trying to
solve problems that the wheel format is *not* intended to cover
(packaging and distribution of non-Python code) is hard - particularly
where we need to express dependencies on such things.

Paul.