[Distutils] Making pip and PyPI work with conda packages

[Chris Barker]

On Tue, May 19, 2015 at 9:15 AM, David Cournapeau <cournape at gmail.com>
wrote:

> Honestly, I still haven't seen a solid explanation of why (at least on
>> Windows) static linking isn't a viable option.
>
>
well - it does get us pretty far....


> Because some libraries simply don't work as static libraries, or are too
> big (MKL comes to mind). Also, we have been historically using static libs
> for our eggs at Enthought on windows, and it has been a nightmare to
> support. It just does not scale when you have 100s of packages.
>

there is also the issue of semi-developers -- I want people to be able to
easily build my package, that depends on a bunch of libs that I really want
to be the same. I suppose I could deliver the static libs themselves, along
with the headers, etc, but that does get ugly.


> But really, once wheels support arbitrary file locations, this becomes
> fairly easy at the packaging level: the remaining issue is one of ABI /
> standards, but that's mostly a non technical issue.
>

yup -- social issues are the big one.


> Gholke has 100s of packages using wheels,
>

doesn't he ship the dlls with the packages, even when  not using static
libs? so that multiple packages may have the same dll? which is almost the
same as a static lib.


> and we ourselves at Enthought have close to 500 packages for windows, all
> packaged as eggs, maybe 30-40 % of which are not python but libs, C/C++
> programs, etc... It is more about agreeing about a common way of doing
> things than a real technical limitation.
>

good to know -- I suspected as much but haven't tried it yet.

If someone were to
>> create and publish a Python compatible static ".lib" file for the
>> various hard-to-build dependencies, extensions could specify that you
>> link with it in setup.py, and all the person building the wheel has to
>> do is download the needed libraries for the build.
>>
>
OK, but from a "social" perspective, this is unlikely to happen (it hasn't
yet!), without some "official" support on PyPi, with pip, or ???

So even if  static is the way to go -- there is an infrastructure need.

If there's a technical reason why dynamic linking at runtime is better
>> than static linking (sufficiently better that it justifies all the
>> effort needed to resolve the issues involved),
>
>
I'm not sure the issues are that much greater -- we could build binary
wheels that hold dynamic libs, and put them up on PyPi -- then other
package maintainers would need to actually use those -- almost the same
thing as getting a variety of package maintainers to use this mythical
repository of static libs.

(and by the way, gcc makes it remarkably had to force a static build)

Another issue I've run  into is nested static libs -- you have to change
yoru setup.py in special ways for that to work:

libnetcdf depends on libhdf5, depends on libcurl and libz and...

getting all that statically linked into my extension is tricky.

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150519/030abcf6/attachment.html>