[Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

Chris Barker chris.barker at noaa.gov
Fri May 15 21:56:01 CEST 2015


On Fri, May 15, 2015 at 1:49 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 14 May 2015 at 19:01, Chris Barker <chris.barker at noaa.gov> wrote:
> > Ah -- here is the issue -- but I think we HAVE pretty much got what we
> need
> > here -- at least for Windows and OS-X. It depends what you mean by
> > "curated", but it seems we have a (defacto?) policy for PyPi: binary
> wheels
> > should be compatible with the python.org builds. So while each package
> wheel
> > is supplied by the package maintainer one way or another, rather than by
> a
> > central entity, it is more or less curated -- or at least standardized.
> And
> > if you are going to put a binary wheel up, you need to make sure it
> matches
> > -- and that is less than trivial for packages that require a third party
> > dependency -- but building the lib statically and then linking it in is
> not
> > inherently easier than doing a dynamic link.
>
> I think the issue is that, if we have 5 different packages that depend
> on (say) libpng, and we're using dynamic builds, then how do those
> packages declare that they need access to libpng.dll?


this is the missing link -- it is a binary build dependency, not a package
dependency -- so not such much that matplotlib-1.4.3 depends on libpng.x.y,
but that:


matplotlib-1.4.3-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl

depends on:

libpng-x.y

(all those binary parts will come from the platform)

That's what's missing now.

And on Windows,
> where does the user put libpng.dll so that it gets picked up?


Well, here is the rub -- Windows dll hell really  is hell -- but I think it
goes into the python dll searchpath (sorry, not on a
Windows box where I can really check this out right now), it can work -- I
know have an in-house product that has multiple python modules sharing a
single dll somehow....



> And how
> does a non-expert user do this ("put it in $DIRECTORY, update your
> PATH, blah blah blah" doesn't work for the average user)?
>

That's why we may need to update the tooling to handle this -- I"m not
totally sure if the current wheel format can support this on Windows --
though it can on OS-X.

In particular, on Windows, note that the shared DLL must either be in
> the directory where the executable is located (which is fun when you
> have virtualenvs, embedded interpreters, etc), or on PATH (which has
> other implications - suppose I have an incompatible version of
> libpng.dll, from mingw, say, somewhere earlier on PATH).
>

that would be dll hell, yes.....


> The problem isn't so much defining a standard ABI that shared DLLs
> need - as you say, that's a more or less solved problem on Windows -
> it's managing how those shared DLLs are made available to Python
> extensions. And *that* is what Unix package managers do for you, and
> Windows doesn't have a good solution for (other than "bundle all the
> dependent DLLs with the app, or suffer DLL hell").
>

exactly -- but if we consider the python install to be the "app", rather
than an individual python bundle, then we _may_ be OK.

PS For a fun exercise, it might be interesting to try breaking conda -
>

Windows really is simply broken [1] in this regard -- so I'm quite sure you
could break conda -- but it does seem to do a pretty good job of not being
broken easily by common uses -- I can't say I know enough about Windows dll
finding or conda to know how...

Oh, and conda is actually broken in this regard on OS-X at this point -- if
you compile your own extension in an anaconda environment, it will find a
shared lib at compile time that it won't find at run time. -- the conda
install process fixes these, but that's a pain when under development --
i.e. you don't want to have to actually install the package with conda to
run a test each time you re-build the dll.. (or even change a bit of python
code...)

But in short -- I'm pretty sure there is a way, on all systems, to have a
standard way to build extension modules, combined with a standard way to
install shared libs, so that a lib can be shared among multiple packages.
So the question remains:

Is there any point? or is the current approach of statically linking all
third party libs the way to go?

If so, then is there any chance of getting folks to conform to this
standard for PyPi hosted binary packages anyway? i.e. the curation problem.

Personally, I'm on the fence here -- I really want newbies to be able to
simply "pip install" as many packages as possible and get a good result
when they do it.

On the other hand, I've found that conda better supports this right now, so
it's easier for me to simply use that for my tools.


-Chris


[1] My take on dll hell:

a) it's inherently difficult -- which is why Linux provides a system
package manager.

b) however, Windows really does make it MORE difficult than it has to be:
  i) it looks first next the executable
  ii) it also looks on the PATH (rather than a separate DLL_PATH)
  Combine these two, and you have some folks dropping dlls next to their
executable, which means they have inadvertently dropped it on the DLL
search path for other apps to find it.

Add to this the (very odd to me) long standing tradition of not putting
extensive version numbers in dll file names, and presto: dll hell!





-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150515/c3300714/attachment-0001.html>


More information about the Distutils-SIG mailing list