On 3 Sep 2014, at 12:24 PM, Reinout van Rees wrote:

b) Making a bunch of wheels seems like a nice solution. Then you can just use a virtualenv and "pip install numpy gdal psycopg2...". But how do you differentiate between ubuntu versions? Not every wheel will work with both 12.04 and 14.04, I'd say. But both ubuntu versions will have the same "linux_x86_64" tag.

=> What is the best way to differentiate here? Separate company-internal "wheelhouse" per ubuntu version? Custom tags?

As a related but alternative suggestion, you could also use [stdeb](https://github.com/astraw/stdeb) either to: 1) build .deb files and host your own repositories — one for 12.04 and one for 14.04 — with mini-dinstall or 2) on each target machine do "pypi-install package-name”. Option #1 is what we do in my lab with the repository at https://debs.strawlab.org/ . -Andrew

On Wed, Sep 03, 2014 at 12:24:10PM +0200, Reinout van Rees wrote:

I'm investigating some options for making our servers a bit more neat. Basic problem: lots of what we do needs mapnik, numpy, gdal, psycopg2 and so. Python libraries with C code and system dependencies.

All of them have ubuntu packages, but especially for gdal we sometimes need a newer version. A PPA can help here, but I thought "a wheel could be nice, too".

System packages? Yes, we use buildout with "syseggrecipe". You pass syseggrecipe a bunch of packages ("mapnik, gdal"), it looks up those packages in the OS and installs them in buildout's "develop-eggs/" directory. Works quite well. Isolation + selective use of system packages.

Do you use buildout 1.x then? buildout 2.x doesn't support isolation, so all system packages are available (unless you wrap it with a virtualenv).

Two questions:

a) If I use ubuntu packages, I'll have to run pip/virtualenv with --system-site-packages. "pip install numpy" will find the global install just fine. But "pip freeze" will give me all site packages' versions, which is not what I want.

=> is there a way to *selectively* use such a system package in an otherwise-isolated virtualenv?

You can symlink selected directories (the Python package directory and sometimes .so files from other subdirs) into the virtualenv. This is hacky and I know of no tool to automate it.

b) Making a bunch of wheels seems like a nice solution. Then you can just use a virtualenv and "pip install numpy gdal psycopg2...". But how do you differentiate between ubuntu versions? Not every wheel will work with both 12.04 and 14.04, I'd say. But both ubuntu versions will have the same "linux_x86_64" tag.

=> What is the best way to differentiate here? Separate company-internal "wheelhouse" per ubuntu version? Custom tags?

My gut feeling says go with a company-internal wheelhouse. A simple directory with a bunch of *.whl files exposed via Apache's mod_autoindex suffices. export PIP_FIND_LINKS=https://intranet.server/wheels/trusty/ to make use of it automatically. $DISTRIB_CODENAME, defined by sourcing /etc/lsb-release, can be helpful here. Marius Gedminas -- Life begins when you can spend your spare time programming instead of watching television. -- Cal Keegan

On 03-09-14 14:22, Marius Gedminas wrote:

Do you use buildout 1.x then? buildout 2.x doesn't support isolation, so all system packages are available (unless you wrap it with a virtualenv).

Buildout 2.x. syseggrecipe basically tells buildout that some package is available globally (by adding a develop-eggs/the_package.egg-link file or so). Otherwise buildout will download and compile numpy/psycopg2/mapnik if it encounters such a dependency in the setup.py.

...

=> is there a way to *selectively* use such a system package in an otherwise-isolated virtualenv?

You can symlink selected directories (the Python package directory and sometimes .so files from other subdirs) into the virtualenv. This is hacky and I know of no tool to automate it.

I was looking for an automated non-hacky way :-) Good to know I haven't missed anything. I'm almost exclusively using buildout, so I'm not that knowledgeable regarding pip.

...

=> What is the best way to differentiate here? Separate company-internal "wheelhouse" per ubuntu version? Custom tags?

My gut feeling says go with a company-internal wheelhouse. A simple directory with a bunch of *.whl files exposed via Apache's mod_autoindex suffices. export PIP_FIND_LINKS=https://intranet.server/wheels/trusty/ to make use of it automatically. $DISTRIB_CODENAME, defined by sourcing /etc/lsb-release, can be helpful here.

Ok, sounds fine. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout@vanrees.org http://www.nelen-schuurmans.nl/ "Learning history by destroying artifacts is a time-honored atrocity"

Another option (along the lines of conda) is pex, which zips up you code + dependencies into a single, zipped executable. https://github.com/pantsbuild/pex Pex has been relatively nice for us, as we can bundle our applications into (mainly) hermetically-sealed binaries, which works well on machines that may have separate system dependencies compared to the applications that run on them. We have a company-internal warehouse that we upload compiled eggs/wheels to, and use the pants build tool to resolve dependencies at build-time http://pantsbuild.github.io/python-readme.html to pull them in. On Wed, Sep 3, 2014 at 7:51 AM, Chris Barker - NOAA Federal < chris.barker@noaa.gov> wrote:

Your might want to consider conda and conda environments for this.

http://www.continuum.io/blog/conda

It provides a single packaging solution for both python and dependencies. And there are probably already recipes for everything you need.

-Chris

...

On Sep 3, 2014, at 3:24 AM, Reinout van Rees wrote:

Hi,

I'm investigating some options for making our servers a bit more neat. Basic problem: lots of what we do needs mapnik, numpy, gdal, psycopg2 and so. Python libraries with C code and system dependencies.

All of them have ubuntu packages, but especially for gdal we sometimes need a newer version. A PPA can help here, but I thought "a wheel could be nice, too".

System packages? Yes, we use buildout with "syseggrecipe". You pass syseggrecipe a bunch of packages ("mapnik, gdal"), it looks up those packages in the OS and installs them in buildout's "develop-eggs/" directory. Works quite well. Isolation + selective use of system packages.

Two questions:

a) If I use ubuntu packages, I'll have to run pip/virtualenv with --system-site-packages. "pip install numpy" will find the global install just fine. But "pip freeze" will give me all site packages' versions, which is not what I want.

=> is there a way to *selectively* use such a system package in an otherwise-isolated virtualenv?

b) Making a bunch of wheels seems like a nice solution. Then you can just use a virtualenv and "pip install numpy gdal psycopg2...". But how do you differentiate between ubuntu versions? Not every wheel will work with both 12.04 and 14.04, I'd say. But both ubuntu versions will have the same "linux_x86_64" tag.

=> What is the best way to differentiate here? Separate company-internal "wheelhouse" per ubuntu version? Custom tags?

Reinout

-- Reinout van Rees http://reinout.vanrees.org/ reinout@vanrees.org http://www.nelen-schuurmans.nl/ "Learning history by destroying artifacts is a time-honored atrocity"

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

---

Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On Thu, Sep 4, 2014 at 9:58 AM, Barry Warsaw wrote:

On Sep 03, 2014, at 12:24 PM, Reinout van Rees wrote:

...

All of them have ubuntu packages, but especially for gdal we sometimes need a newer version. A PPA can help here, but I thought "a wheel could be nice, too".

In many cases, it mostly takes an interested person to get Ubuntu and Debian packages updated.

wouldn't that only update it for the *next* release of debian/ubuntu? maybe you can clarify if/when a package can be updated so that it appears as an update for your current distro version? I roughly have the idea that it's only for security issues. thanks Marcus

On 04-09-14 19:54, Barry Warsaw wrote:

On Sep 04, 2014, at 10:39 AM, Marcus Smith wrote:

...

...

wouldn't that only update it for the*next* release of debian/ubuntu? Generally yes. There's also backports, but that's more effort.

https://help.ubuntu.com/community/UbuntuBackports https://wiki.debian.org/Backports

For my usecase it is mostly that I have a colleague that keeps doing absolute magic with gdal. And when he says "this calculation can be done with 30% of the effort if we use the one-month-old gdal version", then you need *very* up to date ubuntu packages :-) We did use (for this case) the ubuntugis-stable PPA, but that broke a number of our servers somewhere in May or so as it also updated some other dependency which caused breakage. So... that's why Wheels started to sound nice. And compiling wheels yourself and placing them on a server in a directory with various wheels for a specific distribution... Sounds like the most standard option right now. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout@vanrees.org http://www.nelen-schuurmans.nl/ "Learning history by destroying artifacts is a time-honored atrocity"

On 05-09-14 10:00, Andrew Straw wrote:

On 5 Sep 2014, at 9:52 AM, Reinout van Rees wrote:

...

So... that's why Wheels started to sound nice. And compiling wheels yourself and placing them on a server in a directory with various wheels for a specific distribution... Sounds like the most standard option right now.

I haven’t tried it myself but this may also be interesting: https://github.com/spotify/dh-virtualenv

Agreed, looks interesting. I watched the youtube video of the europython Berlin talk about it. In a way it was what caused my original question :-) Why? With dh-virtualenv you can have a debian package with debian dependencies and a virtualenv all ready to go. So... what do you do with debian-level dependencies and how do you tell pip you've got them? Or can you perhaps easily use wheels and get those into the virtualenv and thus into the debian package? So I started thinking and started asking :-) My current thinking is as follows: - One or two basic ubuntu-based boxes: basic ubuntu with a special custom package that pre-installs necessities such as memcached, libjpeg and postgres bindings. - Wheels created for those one or two basic boxes, in a directory per box-type. This way you have some machinery in just one place where you can create wheels at will for your boxes. - Regular pip (or buildout once it supports wheels, I haven't checked yet if it does) to manage the python packages. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout@vanrees.org http://www.nelen-schuurmans.nl/ "Learning history by destroying artifacts is a time-honored atrocity"