[Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

[David Cournapeau]

On Sat, May 16, 2015 at 4:56 AM, Chris Barker <chris.barker at noaa.gov> wrote:

> On Fri, May 15, 2015 at 1:49 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>
>> On 14 May 2015 at 19:01, Chris Barker <chris.barker at noaa.gov> wrote:
>> > Ah -- here is the issue -- but I think we HAVE pretty much got what we
>> need
>> > here -- at least for Windows and OS-X. It depends what you mean by
>> > "curated", but it seems we have a (defacto?) policy for PyPi: binary
>> wheels
>> > should be compatible with the python.org builds. So while each package
>> wheel
>> > is supplied by the package maintainer one way or another, rather than
>> by a
>> > central entity, it is more or less curated -- or at least standardized.
>> And
>> > if you are going to put a binary wheel up, you need to make sure it
>> matches
>> > -- and that is less than trivial for packages that require a third party
>> > dependency -- but building the lib statically and then linking it in is
>> not
>> > inherently easier than doing a dynamic link.
>>
>> I think the issue is that, if we have 5 different packages that depend
>> on (say) libpng, and we're using dynamic builds, then how do those
>> packages declare that they need access to libpng.dll?
>
>
> this is the missing link -- it is a binary build dependency, not a package
> dependency -- so not such much that matplotlib-1.4.3 depends on libpng.x.y,
> but that:
>
>
>
> matplotlib-1.4.3-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
>
> depends on:
>
> libpng-x.y
>
> (all those binary parts will come from the platform)
>
> That's what's missing now.
>
> And on Windows,
>> where does the user put libpng.dll so that it gets picked up?
>
>
> Well, here is the rub -- Windows dll hell really  is hell -- but I think
> it goes into the python dll searchpath (sorry, not on a
> Windows box where I can really check this out right now), it can work -- I
> know have an in-house product that has multiple python modules sharing a
> single dll somehow....
>
>
>
>> And how
>> does a non-expert user do this ("put it in $DIRECTORY, update your
>> PATH, blah blah blah" doesn't work for the average user)?
>>
>
> That's why we may need to update the tooling to handle this -- I"m not
> totally sure if the current wheel format can support this on Windows --
> though it can on OS-X.
>
> In particular, on Windows, note that the shared DLL must either be in
>> the directory where the executable is located (which is fun when you
>> have virtualenvs, embedded interpreters, etc), or on PATH (which has
>> other implications - suppose I have an incompatible version of
>> libpng.dll, from mingw, say, somewhere earlier on PATH).
>>
>
> that would be dll hell, yes.....
>
>
>> The problem isn't so much defining a standard ABI that shared DLLs
>> need - as you say, that's a more or less solved problem on Windows -
>> it's managing how those shared DLLs are made available to Python
>> extensions. And *that* is what Unix package managers do for you, and
>> Windows doesn't have a good solution for (other than "bundle all the
>> dependent DLLs with the app, or suffer DLL hell").
>>
>
> exactly -- but if we consider the python install to be the "app", rather
> than an individual python bundle, then we _may_ be OK.
>
> PS For a fun exercise, it might be interesting to try breaking conda -
>>
>
> Windows really is simply broken [1] in this regard -- so I'm quite sure
> you could break conda -- but it does seem to do a pretty good job of not
> being broken easily by common uses -- I can't say I know enough about
> Windows dll finding or conda to know how...
>
> Oh, and conda is actually broken in this regard on OS-X at this point --
> if you compile your own extension in an anaconda environment, it will find
> a shared lib at compile time that it won't find at run time. -- the conda
> install process fixes these, but that's a pain when under development --
> i.e. you don't want to have to actually install the package with conda to
> run a test each time you re-build the dll.. (or even change a bit of python
> code...)
>
> But in short -- I'm pretty sure there is a way, on all systems, to have a
> standard way to build extension modules, combined with a standard way to
> install shared libs, so that a lib can be shared among multiple packages.
> So the question remains:
>

There is actually no way to do that on windows without modifying the
interpreter somehow. This was somehow discussed a bit at PyCon when talking
about windows packaging:

 1. the simple way to share DLLs across extensions is to put them in the
%PATH%, but that's horrible.
 2. there are ways to put DLLs in a shared directory *not* in the %PATH%
since at least windows XP SP2 and above, through the SetDllDirectory API.

With 2., you still have the issue of DLL hell, which may be resolved
through naming and activation contexts. I had a brief chat with Steve where
he mentioned that this may be a solution, but he was not 100 % sure IIRC.
The main drawback of this solution is that it won't work when inheriting
virtual environments (as you can only set a single directory).

FWIW, we are about to deploy 2. @ Enthought (where we control the python
interpreter, so it is much easier for us).

David


> Is there any point? or is the current approach of statically linking all
> third party libs the way to go?
>
> If so, then is there any chance of getting folks to conform to this
> standard for PyPi hosted binary packages anyway? i.e. the curation problem.
>
> Personally, I'm on the fence here -- I really want newbies to be able to
> simply "pip install" as many packages as possible and get a good result
> when they do it.
>
> On the other hand, I've found that conda better supports this right now,
> so it's easier for me to simply use that for my tools.
>
>
> -Chris
>
>
> [1] My take on dll hell:
>
> a) it's inherently difficult -- which is why Linux provides a system
> package manager.
>
> b) however, Windows really does make it MORE difficult than it has to be:
>   i) it looks first next the executable
>   ii) it also looks on the PATH (rather than a separate DLL_PATH)
>   Combine these two, and you have some folks dropping dlls next to their
> executable, which means they have inadvertently dropped it on the DLL
> search path for other apps to find it.
>
> Add to this the (very odd to me) long standing tradition of not putting
> extensive version numbers in dll file names, and presto: dll hell!
>
>
>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
>
> Chris.Barker at noaa.gov
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150516/7512b5e9/attachment-0001.html>