[Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

[Paul Moore]

On 14 May 2015 at 19:01, Chris Barker <chris.barker at noaa.gov> wrote:
> Ah -- here is the issue -- but I think we HAVE pretty much got what we need
> here -- at least for Windows and OS-X. It depends what you mean by
> "curated", but it seems we have a (defacto?) policy for PyPi: binary wheels
> should be compatible with the python.org builds. So while each package wheel
> is supplied by the package maintainer one way or another, rather than by a
> central entity, it is more or less curated -- or at least standardized. And
> if you are going to put a binary wheel up, you need to make sure it matches
> -- and that is less than trivial for packages that require a third party
> dependency -- but building the lib statically and then linking it in is not
> inherently easier than doing a dynamic link.

I think the issue is that, if we have 5 different packages that depend
on (say) libpng, and we're using dynamic builds, then how do those
packages declare that they need access to libpng.dll? And on Windows,
where does the user put libpng.dll so that it gets picked up? And how
does a non-expert user do this ("put it in $DIRECTORY, update your
PATH, blah blah blah" doesn't work for the average user)?

In particular, on Windows, note that the shared DLL must either be in
the directory where the executable is located (which is fun when you
have virtualenvs, embedded interpreters, etc), or on PATH (which has
other implications - suppose I have an incompatible version of
libpng.dll, from mingw, say, somewhere earlier on PATH).

The problem isn't so much defining a standard ABI that shared DLLs
need - as you say, that's a more or less solved problem on Windows -
it's managing how those shared DLLs are made available to Python
extensions. And *that* is what Unix package managers do for you, and
Windows doesn't have a good solution for (other than "bundle all the
dependent DLLs with the app, or suffer DLL hell").

Paul

PS For a fun exercise, it might be interesting to try breaking conda -
find a Python extension which uses a shared DLL, and check that it
works. Then grab an incompatible copy of that DLL (say a 32-bit
version on a 64-bit system) and try hacking around with PATH, putting
the incompatible DLL in a directory earlier on PATH than the correct
one, in the Windows directory, use an embedded interpreter like
mod_wsgi, tricks like that. If conda survives that, then the solution
that they use might be something worth documenting and might offer an
approach to solving the issue I described above. If it *doesn't*
survive, then that probably implies that the general environment pip
has to work in is less forgiving than the curated environment conda
manages (which is, of course, the whole point of using conda - to get
that curated environment :-))