[Pythonmac-SIG] Advice wanted on dependency building...

[Ronald Oussoren]

On 22 May, 2013, at 19:30, Chris Barker - NOAA Federal <chris.barker at noaa.gov> wrote:

> Hey folks,
> 
> I'm looking for advice, and maybe even some consensus among the
> MacPython community, on how to build and distribute packages with
> non-python dependencies.
> 
> As we all know, a number of Python packages require libs that are used
> outside of python itself. These libs fall into (sort of) what I would
> call two catagories;
> 
> 1) general purpose libs used by multiple python packages: libpng, freetype, etc.
> (I'm still confused on why Apple doesn't provide all these --
> particularly libpng -- what's up with that?)

In general Apple doesn't ship libraries that it doesn't use itself, and for image format support they have their own libraries (in particular the ImageIO framework).  

To move back onto topic, not relying on unix-level libraries in OSX is in a good thing as it makes it easier to support multiple OSX versions with a single set of binaries.


> 
> 2) More specific libs, likely only used by a single python package --
> netcdf, proj.4, etc.
> 
> Users also fall into two categories:
> 
> 1) Folks that do Python development on OS-X much like Linux, etc --
> these folks are likely to use macports or homebrew, or are used to the
> .configure, make, make install dance. We don't need to do anything to
> support these folks -- "pip install" generally works for them.

Except for a number of more complicated libraries (such as PIL/Pillow) when using universal binaries (when using 'pip install', homebrew/macports/... have their own mechanisms for building).

> 
> 2) folks that want to use a Mac like a Mac, and people that develop
> for those folks --  these people need binary installers, and may want
> to be able to use and deploy either packages or applications (Py2app)
> that will run on systems older than the one developed on, or want
> universal builds, or ???
>  - These are the folks I'd like to support, but I'm still unsure as
> to how best to do that.

It would be nice to have a set of binary "packages", based on a reproducable build system.

> 
> Way back when Bob Ippolito maintained a repository of binary packages
> for the mac -- it was a great resource, but he's long since moved on
> to other things.

The binary packages that Bob maintained had IMHO two major problems:

1) The largest problem is that the packages were AFAIK created ad-hoc (Bob or some other contributor did the magic incantations to build library dependencies)

2) The packages were Installer.app packages. The current state of the art for development/project environments is to use virtualenv or buildout to create separated python installations and install all project depedencies there instead of the global site-packages directory. That's not something that's easily supported with Installer.app packages.


> 
> We kind of get away with it because a number of major package
> maintainers have done a good job of providing binaries themselves
> (wxPython, numpy/scipy/matplotlib), but others fail to do so (PIL).
> Plus some of us have minor packages that we want to distribute.
> 
> I'd like to put together an archive, much like Bob's was, or like
> Christoph Gohlke's one for Windows
> (By the way -- that one is HUGE -- I have no idea how he keeps it up!
> http://www.lfd.uci.edu/~gohlke/pythonlibs/)
> 
> But with or without that archive, I still need to build the darn
> things. So now on to the question:
> 
> How should the dependencies be distributed?
> 
> 1) They should be built to match the Python binary being targeted
> (honestly, I think that's now only the Intel 32-64 bit ones -- PPC
> machines, and pre 10.6, are getting really rare...)

No objections there. Supporting PPC is getting way to hard now that
the Xcode version that you can install on recent machines cannot generate
PPC code.  I have VMs with older OSX releases for running the compiler,
but that's barely usable.

> 
> 2) Static or dynamic?
> 
> IIUC, most successful binary packages for the Mac have relied on
> statically linking the dependencies -- this works, and is pretty
> robust. However, it can be kind of a pain to do (though I've finally
> figure how to do it more reliably!). Also, it seems like a waste to me
> for packages that use common dependencies -- how many copies of libpng
> do I really want linked into my single instance of Python at run time?

As long as the libpng state isn't shared static linking isn't really a
problem. It does get a problem when two extensions use the same external
library and share state, for example the curses and curses_panel extensions
in the stdlib.

Dynamic linking has at least two disadvantages:

1) Relocation of the installation prefix is harder due to the way the dynamic linker on OSX looks for libraries: when extension foo.so uses library libfoo.dylib the absolute path of libfoo.dylib is included in the MachO header for foo.so. The header is easily updated using macholib, but that makes installation harder and isn't supported by the standard packaging tools (easy_install and pip)

2) The primary use case for dynamic linking is to share dylibs between extensions, and when those extensions are in different PyPI packages the packaging story gets more complicated. The easiest workaround is to ignore sharing dylibs and still bundle multipe copies of libpng if two different PyPI packages both link with libpng.


> 
> But if dynamic, where do you put them? We'll still want to ship them
> with the binary, so people have a one-click install. I don't think we
> want to install them into a standard location, like /usr/local, as
> that could stomp on something else the user is doing. So:
> - Do we put the in the Python Framework, say in:
> /Library/Frameworks/Python.framework/Versions/2.7/lib
>     - This make some sense, but then you couldn't share them between,
> say, python 2.7 and 3.3 (and however many other versions...
>  - Do we create a new Framework, say:
> /Library/Frameworks/PythonDeps.framework and put them all there? But
> this may confuse things for different deployment targets.

A new framework isn't necessary. There are three locations that could easily be used:

1) A directory in Python.framework, for example /Library/Frameworks/Python.framework/Frameworks

2) A directory in /Library/Python, for example /Library/Python/Externals

3) As 2), but in the users home directory (~/Library/Python/Externals)

The latter is the only one where you can install without admin privileges.

> 
> If we go one of these routes, would we have a separate installer for
> the libs, and all the other installers would depend on that? Or would
> each installer put a copy of the libs it needed into the common
> location, maybe writing over one that's already there (which should be
> OK -- it should be compatible, or have a different version number,
> etc.)

The folks over on distutils-sig are working towards support for wheels (PEP 427, <http://www.python.org/dev/peps/pep-0427/>) at least in pip and distribute/setuptools and possibly in the stdlib as well (for 3.4). It would be nice if the OSX package collection would be in wheel format, that would make it relatively easy to install the packages using the defacto standard tools. 

What I haven't looked into yet is how easy it would be to configure pip to look for packages on PyPI and then look for binaries (wheels) in some other location.

> 
> Note that I've used the term "we" here ;-)  I'm hoping that others
> will join me in following a convention and getting stuff out there,
> but even if not, I'd love feedback on how best to do it.

Good luck :-).  This is fairly boring low-level packaging work, and that tends to put off people. At least it is a lot easier than trying to fix or replace distutils, that has burned out at least 3 generations of developers ;-/

> 
> By the way, the other goal is to build scripts that do the builds the
> way we need for various libs, packages, etc, so that it's easy to do
> it all when new builds are required...
> (maybe use gattai? -- http://sourceforge.net/projects/gattai/)

I haven't used gattai before, but at least it is Python code. Building tools for this from scratch would also not be very hard, I already done so a number of times (such as the build-installer.py script in the CPython repository).

The hard part should be setting up the build infrastructure and build scripts, once couple of relatively hard packages like PIL or wx have been processed adding more packages and new versions of packages should be easy.

Ronald


> 
> Feedback please!!
> 
> -Chris
> 
> -- 
> 
> Christopher Barker, Ph.D.
> Oceanographer
> 
> Emergency Response Division
> NOAA/NOS/OR&R            (206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115       (206) 526-6317   main reception
> 
> Chris.Barker at noaa.gov
> _______________________________________________
> Pythonmac-SIG maillist  -  Pythonmac-SIG at python.org
> http://mail.python.org/mailman/listinfo/pythonmac-sig
> unsubscribe: http://mail.python.org/mailman/options/Pythonmac-SIG