[Distutils] Making pip and PyPI work with conda packages

Chris Barker chris.barker at noaa.gov
Tue May 19 18:40:34 CEST 2015

On Tue, May 19, 2015 at 9:15 AM, David Cournapeau <cournape at gmail.com>

> Honestly, I still haven't seen a solid explanation of why (at least on
>> Windows) static linking isn't a viable option.
well - it does get us pretty far....

> Because some libraries simply don't work as static libraries, or are too
> big (MKL comes to mind). Also, we have been historically using static libs
> for our eggs at Enthought on windows, and it has been a nightmare to
> support. It just does not scale when you have 100s of packages.

there is also the issue of semi-developers -- I want people to be able to
easily build my package, that depends on a bunch of libs that I really want
to be the same. I suppose I could deliver the static libs themselves, along
with the headers, etc, but that does get ugly.

> But really, once wheels support arbitrary file locations, this becomes
> fairly easy at the packaging level: the remaining issue is one of ABI /
> standards, but that's mostly a non technical issue.

yup -- social issues are the big one.

> Gholke has 100s of packages using wheels,

doesn't he ship the dlls with the packages, even when  not using static
libs? so that multiple packages may have the same dll? which is almost the
same as a static lib.

> and we ourselves at Enthought have close to 500 packages for windows, all
> packaged as eggs, maybe 30-40 % of which are not python but libs, C/C++
> programs, etc... It is more about agreeing about a common way of doing
> things than a real technical limitation.

good to know -- I suspected as much but haven't tried it yet.

If someone were to
>> create and publish a Python compatible static ".lib" file for the
>> various hard-to-build dependencies, extensions could specify that you
>> link with it in setup.py, and all the person building the wheel has to
>> do is download the needed libraries for the build.
OK, but from a "social" perspective, this is unlikely to happen (it hasn't
yet!), without some "official" support on PyPi, with pip, or ???

So even if  static is the way to go -- there is an infrastructure need.

If there's a technical reason why dynamic linking at runtime is better
>> than static linking (sufficiently better that it justifies all the
>> effort needed to resolve the issues involved),
I'm not sure the issues are that much greater -- we could build binary
wheels that hold dynamic libs, and put them up on PyPi -- then other
package maintainers would need to actually use those -- almost the same
thing as getting a variety of package maintainers to use this mythical
repository of static libs.

(and by the way, gcc makes it remarkably had to force a static build)

Another issue I've run  into is nested static libs -- you have to change
yoru setup.py in special ways for that to work:

libnetcdf depends on libhdf5, depends on libcurl and libz and...

getting all that statically linked into my extension is tricky.



Christopher Barker, Ph.D.

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150519/030abcf6/attachment.html>

More information about the Distutils-SIG mailing list