[Distutils] Which buildout to use?

Attila Oláh me at aatiis.me
Tue Aug 16 17:04:04 CEST 2011 

On Tue, Aug 16, 2011 at 15:40, Reinout van Rees <reinout at vanrees.org> wrote:
> On 16-08-11 14:41, Attila Oláh wrote:
>>
>> Hello,
>>
>> On Tue, Aug 16, 2011 at 13:45, Reinout van Rees<reinout at vanrees.org>
>>  wrote:
>>>
>>> b) Is anyone actually using 1.5.2 with system packages? And by that I
>>> mean
>>> that you have a dependency on something like numpy and buildout actually
>>> finds that package locally and won't try and grab it off pypi?
>>
>> Yes, we use zc.buildout-1.5.2 with a project that runs Pyramid on
>> Google's App Engine. I run the development environment in Gentoo, and
>> I have net-zope/zope-interface installed in
>> /usr/lib64/python?.?/site-packages/zope/interface. When I run buildout
>> with a system Python that's ABI is supported by the zope-inteface
>> ebuild, it will use the one from the system site-packages. IOW,
>> worksforme.
>
> Just making sure: you have a dependency on zope.interface in your setup.py's
> install_requires? So that buildout actively searches for it?

Good point. It wasn't there, I only listed "pyramid" and a bunch of
other packages. Let's add it and experiment.

Let's update zope-interface first:

$ sudo emerge --update --oneshot zope-interface
[...]
 *
 * Python namespaces:
 *     '/usr/lib64/python2.6/site-packages/zope.interface-3.7.0-py2.6-nspkg.pth':
 *         zope
 *     '/usr/lib64/python3.1/site-packages/zope.interface-3.7.0-py3.1-nspkg.pth':
 *         zope
 *     '/usr/lib64/python2.4/site-packages/zope.interface-3.7.0-py2.4-nspkg.pth':
 *         zope
 *     '/usr/lib64/python2.7/site-packages/zope.interface-3.7.0-py2.7-nspkg.pth':
 *         zope
 *     '/usr/lib64/python3.2/site-packages/zope.interface-3.7.0-py3.2-nspkg.pth':
 *         zope
 *     '/usr/lib64/python2.5/site-packages/zope.interface-3.7.0-py2.5-nspkg.pth':
 *         zope
 *
[...]

Now I update setup.py, adding a few packages just for testing:

$ git clone vemble tmp && cd tmp
$ vim setip.py
$ grep req setup.py
    install_requires=('zope.interface', 'numpy', 'scipy', 'nose', 'six'),

Note that I'll use a simplified buildout.cfg for testing:

$ cat buildout.cfg
[buildout]
develop = .
parts =
    graph
unzip = true

[graph]
recipe = z3c.recipe.depgraph
eggs = vemble
variants = tred
strict = true


Check Python version:

$ python -V
Python 2.7.2+

Note that I have zope.interface, numpy, scipy and nose on my systme. I
do not have 'six' though.

$ python -c "print __import__('zope.interface').interface"
<module 'zope.interface' from
'/usr/lib64/python2.7/site-packages/zope/interface/__init__.pyc'>
$ python -c "print __import__('numpy')"
<module 'numpy' from '/usr/lib64/python2.7/site-packages/numpy/__init__.pyc'>
$ python -c "print __import__('scipy')"
<module 'scipy' from '/usr/lib64/python2.7/site-packages/scipy/__init__.pyc'>
$ python -c "print __import__('nose')"
<module 'nose' from '/usr/lib64/python2.7/site-packages/nose/__init__.pyc'>

$ python bootstrap.py --distribute
[...]
$ ls eggs
zc.buildout-1.5.2-py2.7.egg
$ ./bin/buildout
[...]
Egg from site-packages: distribute 0.6.19
[...]
Getting distribution for 'six'.
Running easy_install:
[...]
Got six 1.0.0.
Picked: six = 1.0.0
[...]
Getting required 'nose'
  required by vemble 0.
We have the best distribution that satisfies 'nose'.
Egg from site-packages: nose 1.1.2
Getting required 'scipy'
  required by vemble 0.
We have the best distribution that satisfies 'scipy'.
Egg from site-packages: scipy 0.9.0
Getting required 'numpy'
  required by vemble 0.
We have the best distribution that satisfies 'numpy'.
Egg from site-packages: numpy 1.6.1
Getting required 'zope.interface'
  required by vemble 0.
We have the best distribution that satisfies 'zope.interface'.
Egg from site-packages: zope.interface 3.7.0

$ ls -1 eggs
six-1.0.0-py2.7.egg
tl.eggdeps-0.4-py2.7.egg
z3c.recipe.depgraph-0.5-py2.7.egg
zc.buildout-1.5.2-py2.7.egg
zc.recipe.egg-1.3.2-py2.7.egg

$ cat bin/graph-graph
[...]

import sys
sys.path[0:0] = [
    '/home/aatiis/repos/tmp/src',
    '/home/aatiis/repos/tmp/eggs/z3c.recipe.depgraph-0.5-py2.7.egg',
    '/home/aatiis/repos/tmp/eggs/tl.eggdeps-0.4-py2.7.egg',
    '/home/aatiis/repos/tmp/eggs/zc.recipe.egg-1.3.2-py2.7.egg',
    '/home/aatiis/repos/tmp/eggs/zc.buildout-1.5.2-py2.7.egg',
    '/usr/lib64/python2.7/site-packages',
    '/usr/lib64/python2.7/site-packages',
    '/usr/lib64/python2.7/site-packages',
    '/usr/lib64/python2.7/site-packages',
    '/usr/lib64/python2.7/site-packages',
    '/home/aatiis/repos/tmp/eggs/six-1.0.0-py2.7.egg',
    ]

[...]

As you can see, it uses '/usr/lib64/python2.7/site-packages' multiple
times, for all the packages that I have on my system.

Note that I used to have some problems with this on Ubuntu, but with
Gentoo it works just fine.

> (ABI compatibility... I *did* upgrade to OSX Lion and installing some things
> give me an "cannot find gcc-4.0" error. But when running my sites, mapnik
> and the other libraries work just fine, so I think that shouldn't be the
> problem. But I don't know what buildout 1.5 does in detail.)

I have the following ABIs enabled:

$ eix zope-interface | grep -i installed
     Installed versions:  3.7.0{tbz2}[1](16:37:53
16/08/11)(python_abis_2.4 python_abis_2.5 python_abis_2.6
python_abis_2.7 python_abis_3.1 python_abis_3.2
-python_abis_2.5-jython -python_abis_2.7-pypy-1.5 -python_abis_3.3)

This basically means that zope.interface is installed in the
site-packages of my Python 2.4 (though it fails with this version),
2.5, 2.6, 2.7, 3.1 and 3.2, and not installed for Python 3.3, Jython
and PyPy.

Also note that Gentoo correctly installs these Python packages as
eggs, including their .egg-info directory.

I hope I could be of some help!

Cheers,

Attila

>> Have you tried the allowed-eggs-from-site-packages [1] option?
>
> Yes. Didn't have any effect, it seemed.
>
>
>
> Reinout
>
> --
> Reinout van Rees                    http://reinout.vanrees.org/
> reinout at vanrees.org             http://www.nelen-schuurmans.nl/
> "If you're not sure what to do, make something. -- Paul Graham"
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>

More information about the Distutils-SIG mailing list