[Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

[Paul Moore]

On 16 May 2015 at 07:35, David Cournapeau <cournape at gmail.com> wrote:
>> But in short -- I'm pretty sure there is a way, on all systems, to have a
>> standard way to build extension modules, combined with a standard way to
>> install shared libs, so that a lib can be shared among multiple packages. So
>> the question remains:
>
> There is actually no way to do that on windows without modifying the
> interpreter somehow. This was somehow discussed a bit at PyCon when talking
> about windows packaging:
>
>  1. the simple way to share DLLs across extensions is to put them in the
> %PATH%, but that's horrible.
>  2. there are ways to put DLLs in a shared directory *not* in the %PATH%
> since at least windows XP SP2 and above, through the SetDllDirectory API.
>
> With 2., you still have the issue of DLL hell, which may be resolved through
> naming and activation contexts. I had a brief chat with Steve where he
> mentioned that this may be a solution, but he was not 100 % sure IIRC. The
> main drawback of this solution is that it won't work when inheriting virtual
> environments (as you can only set a single directory).
>
> FWIW, we are about to deploy 2. @ Enthought (where we control the python
> interpreter, so it is much easier for us).

This is indeed precisely the issue. In general, Python code can run
with "the executable" being in many different places - there are the
standard installs, virtualenvs, and embedding scenarios to consider.
So "put DLLs alongside the executable", which is often how Windows
applications deal with this issue, is not a valid option (that's an
option David missed out above, but that's fine as it doesn't work :-))

Putting DLLs on %PATH% *does* cause problems, and pretty severe ones.
People who use ports of Unix tools, such as myself, hit this a lot -
at one point I got so frustrated with various incompatible versions of
libintl showing up on my PATH, all with the same name, that I went on
a spree of rebuilding all of the GNU tools without libintl support,
just to avoid the issue (and older versions openssl were just as bad
with libeay, etc).

So, as David says, you pretty much have to use SetDllDirectory and
similar features to get a viable location for shared DLLs. I guess it
*may* be possible to call those APIs from a Python extension that you
load *before* using any shared DLLs, but that seems like a very
fragile solution. It's also possible for Python 3.6+ to add a new
"shared DLLs" location for such things, which the core interpreter
includes (either via SetDllDirectory or by the same mechanism that
adds C:\PythonXY\DLLs to the search path at the moment). But that
wouldn't help older versions.

So while I encourage Chris' enthusiasm in looking for a solution to
this issue, I'm not sure it's as easy as he's hoping.

Paul