[Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

[Chris Barker]

On Sat, May 16, 2015 at 4:13 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> > Though it's a lot harder to provide a build environment than just the
> lib to
> > link too .. I"m going to have to think more about that...
>
> It seems to me that the end user doesn't really have a problem here
> ("pip install matplotlib" works fine for me using the existing wheel).
>

Sure -- but that's because Matthew Brett has done a lot of work to make
that happen.

> It's the package maintainers (who have to build the binaries) that
> have the issue because everyone ends up doing the same work over and
> over, building dependencies.


Exactly -- It would be nice if the ecosystem made that easier.


> So rather than trying to address the hard
> problem of dynamic linking, maybe a simpler solution is to set up a
> PyPI-like hosting solution for static libraries of C dependencies?
>
> It could be as simple as a github project that contained a directory
> for each dependency,


I started that here:

https://github.com/PythonCHB/mac-builds

but haven't kept it up. And Matthew Brett has done most of the work here:

https://github.com/MacPython

not sure how he's sharing the static libs -- but it could be done.

 With a setuptools build plugin you could even just
>
specify your libraries in setup.py, and have the plugin download the
> lib files automatically at build time.


actually, that's a pretty cool idea! you'd need  place to host them --
gitHbu is no longer hosting "downloads" are they? though you could probably
use github-pages.. (or somethign else)


> People add libraries to the
> archive simply by posting pull requests. Maybe the project maintainer
> maintains the actual binaries by running the builds separately and
> publishing them separately, or maybe PRs include binaries


or you use a CI system to build them. Something like this is being done by
a bunch of folks for conda/binstar:

https://github.com/ioos/conda-recipes

is just one example.

PS The above is described as if it's single-platform, mostly because I
>
only tend to think about these issues from a Windows POV, but it
> shouldn't be hard to extend it to multi-platform.
>

Indeed -- the MacWheels projects are, of course single platform, but could
be extended. though at the end of the day, there isn't much to share
between building libs on different platforms (unless you are using a
cross-platfrom build tool -- why I was trying out gattai for my stuff)

The conda stuff is multi-platform, though, in fact, you have to write a
separate build script for each platform -- it doesn't really provide
anything to help with that part.

But while these efforts are moving towards removing the need for every
pacakge maintainer to build the deps -- we are now duplicating the effort
of trying to remove duplication of effort :-) -- but maybe just waiting for
something to gain momentum and rise to the top is the answer.

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150516/957e04a3/attachment.html>