[Distutils] Buildout release news

Gary Poster gary.poster at canonical.com
Tue Aug 24 17:47:36 CEST 2010

On Aug 24, 2010, at 10:29 AM, Maurits van Rees wrote:

> Op 24-08-10 13:33, Reinout van Rees schreef:
>> On 08/20/2010 05:22 PM, Gary Poster wrote:
>>> I intend to release zc.buildout 1.5.0 Monday, August 23.
>> I used http://pypi.python.org/pypi/osc.recipe.sysegg/ to reliably grab
>> specific system python packages (PIL, matplotlib, etc). That one is
>> broken with 1.5.0, as it cannot find PIL and friends anymore.
>> Using the
>> http://svn.zope.org/repos/main/zc.buildout/branches/1.4/bootstrap/bootstrap.py
>> helps getting everything working (at least temporary).
>> osc.recipe.sysegg's problem is that a pkg_resources.require(egg) call
>> doesn't find any of the system eggs anymore. Am I correct that that's
>> one of buildout's changes? That the recipes are really totally isolated?

Yes.  Not isolating the recipes makes the process with a system Python very unreliable in my experience.

...maybe we could make it so that recipes could explicitly say that they wanted access to site-packages.  I haven't really thought about it, and if we did that, I certainly would not want to support those sort of recipes further (with system Pythons).  It would be an "at your own risk" sort of thing.

I looked briefly at the recipe.  AFAICT, it would not work with ``virtualenv --no-site-packages`` except for eggs installed in the virtualenv itself (and then it ought to work fine for those, because the new buildout stuff just tries to get out of the way when it encounters the broken -S behavior of ``virtualenv --no-site-packages``).  Therefore, the new buildout functionality should make this recipe unnecessary, and should be more robust in the face of system Python oddities, with automated tests, IIUC (see include-site-packages and allowed-eggs-from-site-packages from http://pypi.python.org/pypi/z3c.recipe.scripts).

How would you like to proceed?

1) We talk about some way of making it so that this kind of recipe can have a "get out of my way" flag to access site-packages.
2) We talk about a way to no longer use this recipe.
3) We do something else you propose.

>> One thing I don't understand is why all my buildouts suddenly started
>> failing: I've pinned zc.buildout in all my buildouts. Did the
>> couple-months-old bootstrap.py (so: no svn:externals!) somehow manage to
>> pull in a new buildout despite the version pin?

In part.  Before the 1.5.0 version, bootstrap would always get the newest buildout, no matter what (unless you used a --version argument to bootstrap).  Then that version of buildout would be responsible for changing to the desired version.

Now, by default, it will get the newest buildout that is final.

>> I had to go through a couple of projects to grab the 1.4 bootstrap.
> If I understand correctly, one of the new things that the new bootstrap.py does, is to actually look at the version pinning for zc.buildout (and probably setuptools/distribute) in buildout.cfg and use that for creating the initial bin/buildout script.  

Close but not quite.  That might be desirable--I considered it--but would have required making yet more of buildout's functionality go into bootstrap.  Bootstrap would have needed to duplicate just enough of the extended-config-parsing behavior of buildout to do the right thing.  That duplication of policy-type code did not seem like a good choice to me.

As above, instead, by default, the new bootstrap gets the newest buildout that is final.

> So if you are using an old bootstrap.py, I would indeed expect that you get the latest zc.buildout version.
> Errr... no, I have now tried it with a few different bootstrap.py versions, and after 'python2.4 bootstrap.py' I always end up with the zc.buildout that is pinned in the versions.  That is not how I remember it, though this suits me fine. :-)  Maybe there is a difference here between using distribute (like I do) or setuptools.˘

Right, pre- and post-1.5.0 you are supposed to *end up* with the desired buildout, irrespective of distribute vs. setuptools.  

(The mechanism, via bootstrap, might have been a bit indirect.  In particular, writing bin/buildout is currently the responsibility of the *original* buildout downloaded by bootstrap.  Changing that behavior, and the similar problem behavior when upgrading, was something I punted on.)

> What I *do* see is that when you start with zc.buildout 1.5.0, then change the pin to 1.4.3 and rerun the buildout, this fails when you are using a buildout extension.  The other way around (first 1.4.3, then change it to 1.5.0) works fine.  I tried it with buildout-versions and with mr.developer as extensions, both having the same result.  For example, start with this, and any bootstrap.py:
> ===================================================
> [buildout]
> extensions = buildout-versions
> buildout_versions_file = buildout.cfg
> parts = pep8
> versions = versions
> [pep8]
> recipe = zc.recipe.egg
> eggs = pep8
> [versions]
> buildout-versions = 1.3
> distribute = 0.6.14
> pep8 = 0.5.0
> zc.buildout = 1.5.0
> zc.recipe.egg = 1.2.2

(I didn't really expect 1.2.2 + 1.5.0 to work, FWIW...zc.buildout 1.5.0 wants zc.recipe.egg 1.3.0 or higher.  There's no way for setup.py to spell "use this version or higher if you you use it at all" AFAIK though.  I guess you report it kinda works though...)

> ===================================================
> Note that I pinned zc.recipe.egg to 1.2.2 as 1.3.2 conflicts with zc.buildout 1.4.3, which I want to try next.

(FWIW, zc.buildout 1.4.4 will automatically do the (over-ridable) pinning of some basic recipes including zc.recipe.egg.)

> Run python2.4 bootstrap.py (probably same effect with python2.anything) and then bin/buildout.  Works fine.  Now change the zc.buildout pin to 1.4.3 and rerun bin/buildout:
> $ bin/buildout
> While:
>  Installing.
>  Loading extensions.
> Error: There is a version conflict.
> We already have: zc.buildout 1.5.0
> With more debugging:
> $ bin/buildout -vvvvvvvvvvvvvvvvvvvvD
> Installing 'buildout-versions'.
> We have the distribution that satisfies 'buildout-versions==1.3'.
> While:
>  Installing.
>  Loading extensions.
> Traceback (most recent call last):
>  File ".../zc.buildout-1.5.0-py2.4.egg/zc/buildout/buildout.py", line 1784, in main
>  File ".../zc.buildout-1.5.0-py2.4.egg/zc/buildout/buildout.py", line 410, in install
>  File ".../zc.buildout-1.5.0-py2.4.egg/zc/buildout/buildout.py", line 960, in _load_extensions
>  File ".../zc.buildout-1.5.0-py2.4.egg/zc/buildout/easy_install.py", line 1085, in install
>  File ".../zc.buildout-1.5.0-py2.4.egg/zc/buildout/easy_install.py", line 942, in install
> VersionConflict: There is a version conflict.
> We already have: zc.buildout 1.5.0
> Starting pdb:
> > .../zc.buildout-1.5.0-py2.4.egg/zc/buildout/easy_install.py(942)install()
> (Pdb) a
> self = <zc.buildout.easy_install.Installer instance at 0x101e1e320>
> specs = ['buildout-versions']
> working_set = <pkg_resources.WorkingSet object at 0x10040e810>
> The workaround is of course to remove bin/buildout, do python2.4 bootstrap.py and run bin/buildout again.  So probably no big deal, but a bit unexpected anyway.

OK, fair enough.  Thanks for the report.

I'm inclined to agree with you that it is not a big deal.  It sounds like you need a resolution to the osc.recipe.sysegg problem more?

> BTW, this is on MacOSX 10.6.4, with a python compiled using this buildout: http://svn.plone.org/svn/collective/buildout/python
> I am normally using virtualenv created by this python, with a few more modules installed, but I tried with a fresh virtualenv and with using the python executable directly as well, with the same results.
> Anyway, I have not seen things go really wrong yet today, and my buildout-using colleagues have not screamed either, so the troubles of the previous beta seem to have been avoided.


> Thanks for your hard work, Gary!



More information about the Distutils-SIG mailing list