[Distutils] wheels or system packages for pip on ubuntu

[Marius Gedminas]

On Wed, Sep 03, 2014 at 12:24:10PM +0200, Reinout van Rees wrote:
> I'm investigating some options for making our servers a bit more
> neat. Basic problem: lots of what we do needs mapnik, numpy, gdal,
> psycopg2 and so. Python libraries with C code and system
> dependencies.
> 
> All of them have ubuntu packages, but especially for gdal we
> sometimes need a newer version. A PPA can help here, but I thought
> "a wheel could be nice, too".
> 
> System packages? Yes, we use buildout with "syseggrecipe". You pass
> syseggrecipe a bunch of packages ("mapnik, gdal"), it looks up those
> packages in the OS and installs them in buildout's "develop-eggs/"
> directory. Works quite well. Isolation + selective use of system
> packages.

Do you use buildout 1.x then?  buildout 2.x doesn't support isolation,
so all system packages are available (unless you wrap it with a
virtualenv).

> Two questions:
> 
> a) If I use ubuntu packages, I'll have to run pip/virtualenv with
> --system-site-packages. "pip install numpy" will find the global
> install just fine. But "pip freeze" will give me all site packages'
> versions, which is not what I want.
> 
> => is there a way to *selectively* use such a system package in an
> otherwise-isolated virtualenv?

You can symlink selected directories (the Python package directory and
sometimes .so files from other subdirs) into the virtualenv.  This is
hacky and I know of no tool to automate it.

> b) Making a bunch of wheels seems like a nice solution. Then you can
> just use a virtualenv and "pip install numpy gdal psycopg2...".  But
> how do you differentiate between ubuntu versions? Not every wheel
> will work with both 12.04 and 14.04, I'd say. But both ubuntu
> versions will have the same "linux_x86_64" tag.
> 
> => What is the best way to differentiate here? Separate
> company-internal "wheelhouse" per ubuntu version? Custom tags?

My gut feeling says go with a company-internal wheelhouse.  A simple
directory with a bunch of *.whl files exposed via Apache's mod_autoindex
suffices.  export PIP_FIND_LINKS=https://intranet.server/wheels/trusty/
to make use of it automatically.  $DISTRIB_CODENAME, defined by sourcing
/etc/lsb-release, can be helpful here.

Marius Gedminas
-- 
Life begins when you can spend your spare time programming instead of
watching television.
                -- Cal Keegan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140903/ee7d4d07/attachment.sig>