Buildout release news
I intend to release zc.buildout 1.5.0 Monday, August 23. I will merge the code from from svn+ssh://svn.zope.org/repos/main/zc.buildout/branches/gary-betafix into trunk. This will change the bootstrap.py script on which some people depend via svn externals. Also, existing bootstrap and buildout invocations that do not specify a particular buildout release will automatically get the new release, as usual. In light of the troubles with the 1.5.0b1 release, particularly for users of virtualenv, I have done four things. 1. I added an automated test for the virtualenv interaction problem encountered in the last release, and fixed it. If you are using virtualenv, it now uses the old behavior for generating scripts. I and others have had success manually testing virtualenv + buildout with with these changes. 2. Today I released zc.buildout 1.4.4 (http://pypi.python.org/pypi/zc.buildout/1.4.4). The 1.4.4 release is a release for people who encounter trouble with the 1.5 line. By switching to the associated bootstrap script (http://svn.zope.org/repos/main/zc.buildout/branches/1.4/bootstrap/bootstrap....) you can stay on 1.4.4 until you are ready to migrate. 3. I have a new help document describing the changes in the 1.5 line. This will be part of the documentation shown on pypi. You can see it here: http://svn.zope.org/zc.buildout/branches/gary-betafix/SYSTEM_PYTHON_HELP.txt... 4. For the future, the new (1.5) versions of Buildout and the bootstrap script now prefer final versions of Buildout, recipes, and extensions. This can be changed by using the --accept-buildout-test-releases flag (or -t for short) when calling bootstrap. This will hopefully allow beta releases of these items to be more easily and safely made in the future. The change log is here: http://svn.zope.org/zc.buildout/branches/gary-betafix/CHANGES.txt?view=marku... Gary
Hi, On Fri, Aug 20, 2010 at 17:22, Gary Poster <gary.poster@canonical.com> wrote:
I intend to release zc.buildout 1.5.0 Monday, August 23. I will merge the code from from svn+ssh://svn.zope.org/repos/main/zc.buildout/branches/gary-betafix into trunk.
Is this the branch that Jim mentioned here: https://bugs.launchpad.net/zc.buildout/+bug/429383/comments/2 ? I still couldn't find a way to generate buildout scripts with excluding the system site-packages. Also, I had troubles with the following bug a while ago, that seems to be fixed in 1.5.0b1 now: - Installing a namespace package using a Python that already has a package in the same namespace (e.g., in the Python's site-packages) failed in some cases. It is now handled correctly. Is there a bootstrap.py file that can be used for early-testing of the release that you're going to make in three days? Attila
On Aug 20, 2010, at 11:38 AM, Attila Oláh wrote:
Hi,
On Fri, Aug 20, 2010 at 17:22, Gary Poster <gary.poster@canonical.com> wrote:
I intend to release zc.buildout 1.5.0 Monday, August 23. I will merge the code from from svn+ssh://svn.zope.org/repos/main/zc.buildout/branches/gary-betafix into trunk.
Is this the branch that Jim mentioned here: https://bugs.launchpad.net/zc.buildout/+bug/429383/comments/2 ?
Yes.
I still couldn't find a way to generate buildout scripts with excluding the system site-packages.
See the documentation I mentioned: http://svn.zope.org/zc.buildout/branches/gary-betafix/SYSTEM_PYTHON_HELP.txt... .
Also, I had troubles with the following bug a while ago, that seems to be fixed in 1.5.0b1 now:
- Installing a namespace package using a Python that already has a package in the same namespace (e.g., in the Python's site-packages) failed in some cases. It is now handled correctly.
Do you mean that you had trouble with this behavior in 1.5.0b1? If so, more details would be helpful, particularly after you try following the instructions in the help file. As you might expect, Debian/Ubuntu compatibility has been the standard I've used for my own experiments and those of my beta testers. Note that, as noted elsewhere, using a system Python is inherently fragile. I've caught about everything I can in the Debian/Ubuntu world, but there are still bugs like https://bugs.edge.launchpad.net/ubuntu/+source/python2.6/+bug/621348 that make things less smooth than they would be otherwise.
Is there a bootstrap.py file that can be used for early-testing of the release that you're going to make in three days?
The bootstrap to use for now is http://svn.zope.org/repos/main/zc.buildout/branches/gary-betafix/bootstrap/b... Next, as I said, read the SYSTEM_PYTHON_HELP.txt I mentioned (though, of course, the gary-betafix bootstrap I mention above is the one to use for now, not the one in trunk). Follow the pertinent instructions, and set your expectations on the basis of its warnings (in particular, the only script-generating recipe that uses the new functionality so far is z3c.recipe.scripts). Now, because things are not yet released, you'll have to do a number of extra steps. These are the ones I follow. Maybe someone else has a better approach, but this tests both buildout *and* bootstrap (other approaches only test buildout AFAIK, and you need both). 1) If you don't have one yet, set up a download-cache per the usual buildout instructions. Generally, the docs are http://pypi.python.org/pypi/zc.buildout#using-a-download-cache . More specifically, I do what is described here: https://dev.launchpad.net/HackingLazrLibraries#Global%20Cache . 2) In your download-cache's dist's directory (~/.buildout/download-cache/dist/ if you followed the Global Cache instructions) put the following four items in: - the setuptools egg for the Python version you are using (from http://pypi.python.org/pypi/setuptools) - temporary releases of the three pertinent releases, available from http://people.canonical.com/~gary/zc.buildout-1.5.0-early-releases/ 3) When you are ready to call the new bootstrap in a fresh checkout of your project, you'll need to specify the download-cache-dist directory as an argument. Here's an example of my call: python bootstrap.py --download-base=/home/gary/.buildout/download-cache/dist/ Now you should be testing the releases as I intend to make them. Note that, if I do have to make changes before that release on the basis of feedback, you will need to remove all versions of these (in download-cache and in any cached eggs) before you get the new changes. Gary
On Aug 20, 2010, at 11:22 AM, Gary Poster wrote:
I intend to release zc.buildout 1.5.0 Monday, August 23. I will merge the code from from svn+ssh://svn.zope.org/repos/main/zc.buildout/branches/gary-betafix into trunk.
Cool. Once you do the merge (or maybe before), what's the best way to test the code on our own projects? -Barry
On Aug 20, 2010, at 11:49 AM, Barry Warsaw wrote:
On Aug 20, 2010, at 11:22 AM, Gary Poster wrote:
I intend to release zc.buildout 1.5.0 Monday, August 23. I will merge the code from from svn+ssh://svn.zope.org/repos/main/zc.buildout/branches/gary-betafix into trunk.
Cool. Once you do the merge (or maybe before), what's the best way to test the code on our own projects?
The email I sent to Attila should have the instructions, painful as they are, for "before". After the release, the http://svn.zope.org/zc.buildout/branches/gary-betafix/SYSTEM_PYTHON_HELP.txt... doc should be what you need, alone. (As I said, that will be on the PyPI page as well.) Gary
I have merged the betafix branch to trunk and made the releases of zc.buildout 1.5.0, zc.recipe.egg 1.3.0, and z3c.recipe.scripts 1.0.0. Gary On Aug 20, 2010, at 11:22 AM, Gary Poster wrote:
I intend to release zc.buildout 1.5.0 Monday, August 23. I will merge the code from from svn+ssh://svn.zope.org/repos/main/zc.buildout/branches/gary-betafix into trunk.
This will change the bootstrap.py script on which some people depend via svn externals. Also, existing bootstrap and buildout invocations that do not specify a particular buildout release will automatically get the new release, as usual.
In light of the troubles with the 1.5.0b1 release, particularly for users of virtualenv, I have done four things.
1. I added an automated test for the virtualenv interaction problem encountered in the last release, and fixed it. If you are using virtualenv, it now uses the old behavior for generating scripts. I and others have had success manually testing virtualenv + buildout with with these changes.
2. Today I released zc.buildout 1.4.4 (http://pypi.python.org/pypi/zc.buildout/1.4.4). The 1.4.4 release is a release for people who encounter trouble with the 1.5 line. By switching to the associated bootstrap script (http://svn.zope.org/repos/main/zc.buildout/branches/1.4/bootstrap/bootstrap....) you can stay on 1.4.4 until you are ready to migrate.
3. I have a new help document describing the changes in the 1.5 line. This will be part of the documentation shown on pypi. You can see it here: http://svn.zope.org/zc.buildout/branches/gary-betafix/SYSTEM_PYTHON_HELP.txt...
4. For the future, the new (1.5) versions of Buildout and the bootstrap script now prefer final versions of Buildout, recipes, and extensions. This can be changed by using the --accept-buildout-test-releases flag (or -t for short) when calling bootstrap. This will hopefully allow beta releases of these items to be more easily and safely made in the future.
The change log is here: http://svn.zope.org/zc.buildout/branches/gary-betafix/CHANGES.txt?view=marku...
Gary
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Hello, On Mon, Aug 23, 2010 at 17:43, Gary Poster <gary.poster@canonical.com> wrote:
I have merged the betafix branch to trunk and made the releases of zc.buildout 1.5.0, zc.recipe.egg 1.3.0, and z3c.recipe.scripts 1.0.0.
Here are some observations. 1. I had a line like this in my ~/.bashrc: export PYTHONPATH="/home/aatiis/local/lib/python2.6/site-packages/" This was causing a "RuntimeError: maximum recursion depth exceeded" when running "python bootstrap.py", with Python 2.[4567]. So I had to remove this line; however, this issue seems to be related to the new bootstrap.py file only, coming from http://svn.zope.org/repos/main/zc.buildout/trunk/bootstrap/bootstrap.py 2. The recipe called "djangorecipe" has a weird issue. The cause seems to be that it uses ``zc.buildout.easy_install.scripts`` instead of ``zc.buildout.easy_install.sitepackage_safe_scripts``. The generated scripts have the following issue: Say I have PIL installed globally, and the generated script will have the global one's project root directory added to sys.path, instead of the one in the "eggs" directory (there *is* a PIL in the "eggs" dir, though). Now, my "./bin/django" looks like this: #!/usr/bin/python2.6 import sys sys.path[0:0] = [ ... '/usr/lib64/python2.6/site-packages/PIL', ... ] ... Why is the "PIL" added to sys.path? Regards, Attila
On Aug 23, 2010, at 6:38 PM, Attila Oláh wrote:
Hello,
On Mon, Aug 23, 2010 at 17:43, Gary Poster <gary.poster@canonical.com> wrote:
I have merged the betafix branch to trunk and made the releases of zc.buildout 1.5.0, zc.recipe.egg 1.3.0, and z3c.recipe.scripts 1.0.0.
Here are some observations.
1. I had a line like this in my ~/.bashrc:
export PYTHONPATH="/home/aatiis/local/lib/python2.6/site-packages/"
This was causing a "RuntimeError: maximum recursion depth exceeded" when running "python bootstrap.py", with Python 2.[4567]. So I had to remove this line; however, this issue seems to be related to the new bootstrap.py file only, coming from http://svn.zope.org/repos/main/zc.buildout/trunk/bootstrap/bootstrap.py
Huh, that's interesting. I was not able to dupe (I tried ``env PYTHONPATH=/home/gary/py/lib/python2.6/site-packages 26python bootstrap.py``, where 26python is a local Python). Could you give me a traceback? (Off-list, perhaps.)
2. The recipe called "djangorecipe" has a weird issue. The cause seems to be that it uses ``zc.buildout.easy_install.scripts`` instead of ``zc.buildout.easy_install.sitepackage_safe_scripts``.
Yeah. That's one we use too, and I intend to upgrade it eventually if no-one beats me to it.
he generated scripts have the following issue: Say I have PIL installed globally, and the generated script will have the global one's project root directory added to sys.path, instead of the one in the "eggs" directory (there *is* a PIL in the "eggs" dir, though). Now, my "./bin/django" looks like this:
#!/usr/bin/python2.6
import sys sys.path[0:0] = [ ... '/usr/lib64/python2.6/site-packages/PIL', ... ] ...
Why is the "PIL" added to sys.path?
As you and I said, it's the old code path, so all bets are off with a system Python. Wanna upgrade djangorecipe for me? :-) Anyway, in this case, I believe that particular spelling of the path has to do with how PIL is often installed. Gary
On Mon, Aug 23, 2010 at 17:43, Gary Poster <gary.poster@canonical.com> wrote:
I have merged the betafix branch to trunk and made the releases of zc.buildout 1.5.0, zc.recipe.egg 1.3.0, and z3c.recipe.scripts 1.0.0.
Awesome! For the next release, can you add a note on each new functionality in the documentation (something like "new in 1.5") ? It's hard to see the changes. Thanks Gary for the release. -- Sebastien Douche <sdouche@gmail.com> Twitter: http://bit.ly/afkrK (agile, lean, python, open source)
On Aug 23, 2010, at 7:13 PM, Sebastien Douche wrote:
On Mon, Aug 23, 2010 at 17:43, Gary Poster <gary.poster@canonical.com> wrote:
I have merged the betafix branch to trunk and made the releases of zc.buildout 1.5.0, zc.recipe.egg 1.3.0, and z3c.recipe.scripts 1.0.0.
Awesome! For the next release, can you add a note on each new functionality in the documentation (something like "new in 1.5") ? It's hard to see the changes.
...maybe... :-) What's the goal? I think that the main things end users ought to know about are these. (1) How does the system Python support work? See http://pypi.python.org/pypi/zc.buildout/1.5.0#system-python-and-zc-buildout-... . It links out to http://pypi.python.org/pypi/z3c.recipe.scripts . (If you really want to get into the nitty gritty you can see http://pypi.python.org/pypi/zc.buildout/1.5.0#the-sitepackage-safe-scripts-f... and the following two sections, but I'm not sure it's worth highlighting. That's maintainer-y information and in my perfect world would not be on the main end-user PyPI page. Sorry, I didn't work on improving the docs. :-) ) (2) How do you get alpha/beta releases of zc.buildout and friends in the future? You can use python bootstrap.py -t or specify a specific version, as mentioned in the CHANGES (http://pypi.python.org/pypi/zc.buildout/1.5.0#id9). I figure that's something that will be emphasized the next time someone wants help testing a beta release, and it doesn't need too much discussion now. (3) Can you use Python 2.7? Yes, see CHANGES (http://pypi.python.org/pypi/zc.buildout/1.5.0#id9). I did other stuff, but it was mostly internal bits related to those things. Well...maybe mentioning the new bootstrap features would be nice (they are mentioned in CHANGES), and mentioning the new output when running buildout with -v that shows you what packages came from site-packages, if any (I forgot to talk about that, though it is tested). Anyway...
Thanks Gary for the release.
Thanks for the thanks. Gary
On 08/20/2010 05:22 PM, Gary Poster wrote:
I intend to release zc.buildout 1.5.0 Monday, August 23.
I used http://pypi.python.org/pypi/osc.recipe.sysegg/ to reliably grab specific system python packages (PIL, matplotlib, etc). That one is broken with 1.5.0, as it cannot find PIL and friends anymore. Using the http://svn.zope.org/repos/main/zc.buildout/branches/1.4/bootstrap/bootstrap.... helps getting everything working (at least temporary). osc.recipe.sysegg's problem is that a pkg_resources.require(egg) call doesn't find any of the system eggs anymore. Am I correct that that's one of buildout's changes? That the recipes are really totally isolated? One thing I don't understand is why all my buildouts suddenly started failing: I've pinned zc.buildout in all my buildouts. Did the couple-months-old bootstrap.py (so: no svn:externals!) somehow manage to pull in a new buildout despite the version pin? I had to go through a couple of projects to grab the 1.4 bootstrap. Reinout -- Reinout van Rees - reinout@vanrees.org - http://reinout.vanrees.org Programmer at http://www.nelen-schuurmans.nl "Military engineers build missiles. Civil engineers build targets"
Op 24-08-10 13:33, Reinout van Rees schreef:
On 08/20/2010 05:22 PM, Gary Poster wrote:
I intend to release zc.buildout 1.5.0 Monday, August 23.
I used http://pypi.python.org/pypi/osc.recipe.sysegg/ to reliably grab specific system python packages (PIL, matplotlib, etc). That one is broken with 1.5.0, as it cannot find PIL and friends anymore.
Using the http://svn.zope.org/repos/main/zc.buildout/branches/1.4/bootstrap/bootstrap.... helps getting everything working (at least temporary).
osc.recipe.sysegg's problem is that a pkg_resources.require(egg) call doesn't find any of the system eggs anymore. Am I correct that that's one of buildout's changes? That the recipes are really totally isolated?
One thing I don't understand is why all my buildouts suddenly started failing: I've pinned zc.buildout in all my buildouts. Did the couple-months-old bootstrap.py (so: no svn:externals!) somehow manage to pull in a new buildout despite the version pin?
I had to go through a couple of projects to grab the 1.4 bootstrap.
If I understand correctly, one of the new things that the new bootstrap.py does, is to actually look at the version pinning for zc.buildout (and probably setuptools/distribute) in buildout.cfg and use that for creating the initial bin/buildout script. So if you are using an old bootstrap.py, I would indeed expect that you get the latest zc.buildout version. Errr... no, I have now tried it with a few different bootstrap.py versions, and after 'python2.4 bootstrap.py' I always end up with the zc.buildout that is pinned in the versions. That is not how I remember it, though this suits me fine. :-) Maybe there is a difference here between using distribute (like I do) or setuptools.˘ What I *do* see is that when you start with zc.buildout 1.5.0, then change the pin to 1.4.3 and rerun the buildout, this fails when you are using a buildout extension. The other way around (first 1.4.3, then change it to 1.5.0) works fine. I tried it with buildout-versions and with mr.developer as extensions, both having the same result. For example, start with this, and any bootstrap.py: =================================================== [buildout] extensions = buildout-versions buildout_versions_file = buildout.cfg parts = pep8 versions = versions [pep8] recipe = zc.recipe.egg eggs = pep8 [versions] buildout-versions = 1.3 distribute = 0.6.14 pep8 = 0.5.0 zc.buildout = 1.5.0 zc.recipe.egg = 1.2.2 =================================================== Note that I pinned zc.recipe.egg to 1.2.2 as 1.3.2 conflicts with zc.buildout 1.4.3, which I want to try next. Run python2.4 bootstrap.py (probably same effect with python2.anything) and then bin/buildout. Works fine. Now change the zc.buildout pin to 1.4.3 and rerun bin/buildout: $ bin/buildout While: Installing. Loading extensions. Error: There is a version conflict. We already have: zc.buildout 1.5.0 With more debugging: $ bin/buildout -vvvvvvvvvvvvvvvvvvvvD Installing 'buildout-versions'. We have the distribution that satisfies 'buildout-versions==1.3'. While: Installing. Loading extensions. Traceback (most recent call last): File ".../zc.buildout-1.5.0-py2.4.egg/zc/buildout/buildout.py", line 1784, in main File ".../zc.buildout-1.5.0-py2.4.egg/zc/buildout/buildout.py", line 410, in install File ".../zc.buildout-1.5.0-py2.4.egg/zc/buildout/buildout.py", line 960, in _load_extensions File ".../zc.buildout-1.5.0-py2.4.egg/zc/buildout/easy_install.py", line 1085, in install File ".../zc.buildout-1.5.0-py2.4.egg/zc/buildout/easy_install.py", line 942, in install VersionConflict: There is a version conflict. We already have: zc.buildout 1.5.0 Starting pdb:
.../zc.buildout-1.5.0-py2.4.egg/zc/buildout/easy_install.py(942)install() (Pdb) a self = <zc.buildout.easy_install.Installer instance at 0x101e1e320> specs = ['buildout-versions'] working_set = <pkg_resources.WorkingSet object at 0x10040e810> The workaround is of course to remove bin/buildout, do python2.4 bootstrap.py and run bin/buildout again. So probably no big deal, but a bit unexpected anyway. BTW, this is on MacOSX 10.6.4, with a python compiled using this buildout: http://svn.plone.org/svn/collective/buildout/python I am normally using virtualenv created by this python, with a few more modules installed, but I tried with a fresh virtualenv and with using the python executable directly as well, with the same results. Anyway, I have not seen things go really wrong yet today, and my buildout-using colleagues have not screamed either, so the troubles of the previous beta seem to have been avoided. Thanks for your hard work, Gary! -- Maurits van Rees Programmer, Zest Software
On Aug 24, 2010, at 10:29 AM, Maurits van Rees wrote:
Op 24-08-10 13:33, Reinout van Rees schreef:
On 08/20/2010 05:22 PM, Gary Poster wrote:
I intend to release zc.buildout 1.5.0 Monday, August 23.
I used http://pypi.python.org/pypi/osc.recipe.sysegg/ to reliably grab specific system python packages (PIL, matplotlib, etc). That one is broken with 1.5.0, as it cannot find PIL and friends anymore.
Using the http://svn.zope.org/repos/main/zc.buildout/branches/1.4/bootstrap/bootstrap.... helps getting everything working (at least temporary).
osc.recipe.sysegg's problem is that a pkg_resources.require(egg) call doesn't find any of the system eggs anymore. Am I correct that that's one of buildout's changes? That the recipes are really totally isolated?
Yes. Not isolating the recipes makes the process with a system Python very unreliable in my experience. ...maybe we could make it so that recipes could explicitly say that they wanted access to site-packages. I haven't really thought about it, and if we did that, I certainly would not want to support those sort of recipes further (with system Pythons). It would be an "at your own risk" sort of thing. I looked briefly at the recipe. AFAICT, it would not work with ``virtualenv --no-site-packages`` except for eggs installed in the virtualenv itself (and then it ought to work fine for those, because the new buildout stuff just tries to get out of the way when it encounters the broken -S behavior of ``virtualenv --no-site-packages``). Therefore, the new buildout functionality should make this recipe unnecessary, and should be more robust in the face of system Python oddities, with automated tests, IIUC (see include-site-packages and allowed-eggs-from-site-packages from http://pypi.python.org/pypi/z3c.recipe.scripts). How would you like to proceed? 1) We talk about some way of making it so that this kind of recipe can have a "get out of my way" flag to access site-packages. 2) We talk about a way to no longer use this recipe. 3) We do something else you propose.
One thing I don't understand is why all my buildouts suddenly started failing: I've pinned zc.buildout in all my buildouts. Did the couple-months-old bootstrap.py (so: no svn:externals!) somehow manage to pull in a new buildout despite the version pin?
In part. Before the 1.5.0 version, bootstrap would always get the newest buildout, no matter what (unless you used a --version argument to bootstrap). Then that version of buildout would be responsible for changing to the desired version. Now, by default, it will get the newest buildout that is final.
I had to go through a couple of projects to grab the 1.4 bootstrap.
If I understand correctly, one of the new things that the new bootstrap.py does, is to actually look at the version pinning for zc.buildout (and probably setuptools/distribute) in buildout.cfg and use that for creating the initial bin/buildout script.
Close but not quite. That might be desirable--I considered it--but would have required making yet more of buildout's functionality go into bootstrap. Bootstrap would have needed to duplicate just enough of the extended-config-parsing behavior of buildout to do the right thing. That duplication of policy-type code did not seem like a good choice to me. As above, instead, by default, the new bootstrap gets the newest buildout that is final.
So if you are using an old bootstrap.py, I would indeed expect that you get the latest zc.buildout version.
Errr... no, I have now tried it with a few different bootstrap.py versions, and after 'python2.4 bootstrap.py' I always end up with the zc.buildout that is pinned in the versions. That is not how I remember it, though this suits me fine. :-) Maybe there is a difference here between using distribute (like I do) or setuptools.˘
Right, pre- and post-1.5.0 you are supposed to *end up* with the desired buildout, irrespective of distribute vs. setuptools. (The mechanism, via bootstrap, might have been a bit indirect. In particular, writing bin/buildout is currently the responsibility of the *original* buildout downloaded by bootstrap. Changing that behavior, and the similar problem behavior when upgrading, was something I punted on.)
What I *do* see is that when you start with zc.buildout 1.5.0, then change the pin to 1.4.3 and rerun the buildout, this fails when you are using a buildout extension. The other way around (first 1.4.3, then change it to 1.5.0) works fine. I tried it with buildout-versions and with mr.developer as extensions, both having the same result. For example, start with this, and any bootstrap.py:
=================================================== [buildout] extensions = buildout-versions buildout_versions_file = buildout.cfg parts = pep8 versions = versions
[pep8] recipe = zc.recipe.egg eggs = pep8
[versions] buildout-versions = 1.3 distribute = 0.6.14 pep8 = 0.5.0 zc.buildout = 1.5.0 zc.recipe.egg = 1.2.2
(I didn't really expect 1.2.2 + 1.5.0 to work, FWIW...zc.buildout 1.5.0 wants zc.recipe.egg 1.3.0 or higher. There's no way for setup.py to spell "use this version or higher if you you use it at all" AFAIK though. I guess you report it kinda works though...)
===================================================
Note that I pinned zc.recipe.egg to 1.2.2 as 1.3.2 conflicts with zc.buildout 1.4.3, which I want to try next.
(FWIW, zc.buildout 1.4.4 will automatically do the (over-ridable) pinning of some basic recipes including zc.recipe.egg.)
Run python2.4 bootstrap.py (probably same effect with python2.anything) and then bin/buildout. Works fine. Now change the zc.buildout pin to 1.4.3 and rerun bin/buildout:
$ bin/buildout While: Installing. Loading extensions. Error: There is a version conflict. We already have: zc.buildout 1.5.0
With more debugging:
$ bin/buildout -vvvvvvvvvvvvvvvvvvvvD Installing 'buildout-versions'. We have the distribution that satisfies 'buildout-versions==1.3'. While: Installing. Loading extensions. Traceback (most recent call last): File ".../zc.buildout-1.5.0-py2.4.egg/zc/buildout/buildout.py", line 1784, in main File ".../zc.buildout-1.5.0-py2.4.egg/zc/buildout/buildout.py", line 410, in install File ".../zc.buildout-1.5.0-py2.4.egg/zc/buildout/buildout.py", line 960, in _load_extensions File ".../zc.buildout-1.5.0-py2.4.egg/zc/buildout/easy_install.py", line 1085, in install File ".../zc.buildout-1.5.0-py2.4.egg/zc/buildout/easy_install.py", line 942, in install VersionConflict: There is a version conflict. We already have: zc.buildout 1.5.0 Starting pdb:
.../zc.buildout-1.5.0-py2.4.egg/zc/buildout/easy_install.py(942)install() (Pdb) a self = <zc.buildout.easy_install.Installer instance at 0x101e1e320> specs = ['buildout-versions'] working_set = <pkg_resources.WorkingSet object at 0x10040e810>
The workaround is of course to remove bin/buildout, do python2.4 bootstrap.py and run bin/buildout again. So probably no big deal, but a bit unexpected anyway.
OK, fair enough. Thanks for the report. I'm inclined to agree with you that it is not a big deal. It sounds like you need a resolution to the osc.recipe.sysegg problem more?
BTW, this is on MacOSX 10.6.4, with a python compiled using this buildout: http://svn.plone.org/svn/collective/buildout/python I am normally using virtualenv created by this python, with a few more modules installed, but I tried with a fresh virtualenv and with using the python executable directly as well, with the same results.
Anyway, I have not seen things go really wrong yet today, and my buildout-using colleagues have not screamed either, so the troubles of the previous beta seem to have been avoided.
Cool.
Thanks for your hard work, Gary!
Welcome. Gary
Op 24-08-10 17:47, Gary Poster schreef:
On Aug 24, 2010, at 10:29 AM, Maurits van Rees wrote:
Op 24-08-10 13:33, Reinout van Rees schreef:
On 08/20/2010 05:22 PM, Gary Poster wrote:
I intend to release zc.buildout 1.5.0 Monday, August 23.
I used http://pypi.python.org/pypi/osc.recipe.sysegg/ to reliably grab specific system python packages (PIL, matplotlib, etc). That one is broken with 1.5.0, as it cannot find PIL and friends anymore.
Using the http://svn.zope.org/repos/main/zc.buildout/branches/1.4/bootstrap/bootstrap.... helps getting everything working (at least temporary).
osc.recipe.sysegg's problem is that a pkg_resources.require(egg) call doesn't find any of the system eggs anymore. Am I correct that that's one of buildout's changes? That the recipes are really totally isolated? (...) $ bin/buildout While: Installing. Loading extensions. Error: There is a version conflict. We already have: zc.buildout 1.5.0
With more debugging:
$ bin/buildout -vvvvvvvvvvvvvvvvvvvvD Installing 'buildout-versions'. We have the distribution that satisfies 'buildout-versions==1.3'. While: Installing. Loading extensions. Traceback (most recent call last): File ".../zc.buildout-1.5.0-py2.4.egg/zc/buildout/buildout.py", line 1784, in main File ".../zc.buildout-1.5.0-py2.4.egg/zc/buildout/buildout.py", line 410, in install File ".../zc.buildout-1.5.0-py2.4.egg/zc/buildout/buildout.py", line 960, in _load_extensions File ".../zc.buildout-1.5.0-py2.4.egg/zc/buildout/easy_install.py", line 1085, in install File ".../zc.buildout-1.5.0-py2.4.egg/zc/buildout/easy_install.py", line 942, in install VersionConflict: There is a version conflict. We already have: zc.buildout 1.5.0 Starting pdb:
.../zc.buildout-1.5.0-py2.4.egg/zc/buildout/easy_install.py(942)install() (Pdb) a self =<zc.buildout.easy_install.Installer instance at 0x101e1e320> specs = ['buildout-versions'] working_set =<pkg_resources.WorkingSet object at 0x10040e810>
The workaround is of course to remove bin/buildout, do python2.4 bootstrap.py and run bin/buildout again. So probably no big deal, but a bit unexpected anyway.
OK, fair enough. Thanks for the report.
I'm inclined to agree with you that it is not a big deal. It sounds like you need a resolution to the osc.recipe.sysegg problem more?
Ah, you confuse me with my evil twin brother Reinout. :-) Well, okay, he is not evil and no twin, but he is my brother. The first part of the mail (about osc.recipe.sysegg) was from him, and I reacted to that in the second half with my own little problems. To me it sounds like osc.recipe.sysegg should not be necessary anymore when using zc.buildout 1.5.0 and zc.recipe.egg 1.3.2. -- Maurits van Rees Programmer, Zest Software
On 08/25/2010 11:21 AM, Maurits van Rees wrote:
Ah, you confuse me with my evil twin brother Reinout. :-) Well, okay, he is not evil and no twin, but he is my brother. The first part of the mail (about osc.recipe.sysegg) was from him, and I reacted to that in the second half with my own little problems.
He he :-) Long live confusion :-) Reinout -- Reinout van Rees - reinout@vanrees.org - http://reinout.vanrees.org Programmer at http://www.nelen-schuurmans.nl "Military engineers build missiles. Civil engineers build targets"
On 08/24/2010 05:47 PM, Gary Poster wrote:
How would you like to proceed?
1) We talk about some way of making it so that this kind of recipe can have a "get out of my way" flag to access site-packages. 2) We talk about a way to no longer use this recipe. 3) We do something else you propose.
Buildout itself supporting system eggs reliably is of course the handiest option, so that would be option 2. It is perhaps handiest if I explain my usecase. I use buildout for my Django projects (with the django recipe being a big piece of the pie). Lots of geographical processing, so mapnik, matplotlib, numpy and a couple of other not-pure-python packages. Some of those are notoriously difficult/impossible to install with a simple "easy_install" or generic buildout dependency install. "aptitude install python-matplotlib" and friends (or the windows equivalent) is the only practical way to get it running. a) I *do* need the system python. b) I prefer listing all dependencies (so also the shitty-to-install ones) in my setup.py. So I need to be able to tell buildout to grab them from the system packages instead of shooting itself in the foot by trying to grab them from pypi. c) I prefer having buildout search explicitly for these specific dependencies and complain when it doesn't find them. Helps with missing libraries. Now, I looked at the new buildout 1.5 system packages options, but I wondered whether it would work fully. Only the script recipe seems to support it at the moment and it didn't look from the documentation that such a globally-found dependency would be valid for the entire buildout (and all the other parts). If I use that osc.recipe.sysegg and list that part first, it makes sure all other parts use the specified system eggs. Just a minute ago, I tried experimenting with a z3c.recipe.egg part mentioned as the first one, but I got an "ImportError: Entry point ('zc.buildout', 'default') not found". I haven't updated the bootstrap yet, so that might be the cause (though that'd be a bug). Reinout -- Reinout van Rees - reinout@vanrees.org - http://reinout.vanrees.org Programmer at http://www.nelen-schuurmans.nl "Military engineers build missiles. Civil engineers build targets"
On Aug 25, 2010, at 5:50 AM, Reinout van Rees wrote:
On 08/24/2010 05:47 PM, Gary Poster wrote:
How would you like to proceed?
1) We talk about some way of making it so that this kind of recipe can have a "get out of my way" flag to access site-packages. 2) We talk about a way to no longer use this recipe. 3) We do something else you propose.
Sorry for not replying sooner. FWIW, including me directly in the distribution list will get faster replies.
Buildout itself supporting system eggs reliably is of course the handiest option, so that would be option 2.
It is perhaps handiest if I explain my usecase. I use buildout for my Django projects (with the django recipe being a big piece of the pie). Lots of geographical processing, so mapnik, matplotlib, numpy and a couple of other not-pure-python packages.
Some of those are notoriously difficult/impossible to install with a simple "easy_install" or generic buildout dependency install. "aptitude install python-matplotlib" and friends (or the windows equivalent) is the only practical way to get it running.
We have similar requirements.
a) I *do* need the system python.
b) I prefer listing all dependencies (so also the shitty-to-install ones) in my setup.py. So I need to be able to tell buildout to grab them from the system packages instead of shooting itself in the foot by trying to grab them from pypi.
c) I prefer having buildout search explicitly for these specific dependencies and complain when it doesn't find them. Helps with missing libraries.
Now, I looked at the new buildout 1.5 system packages options, but I wondered whether it would work fully.
As I said, the requirements sound the same to me.
Only the script recipe seems to support it at the moment and it didn't look from the documentation that such a globally-found dependency would be valid for the entire buildout (and all the other parts). If I use that osc.recipe.sysegg and list that part first, it makes sure all other parts use the specified system eggs.
But then you are susceptible to many problems that occur in system Pythons that I've been debugging for this release. Which django recipe do you use, by the way? I intend to update djangorecipe, using some changes that Attila made.
Just a minute ago, I tried experimenting with a z3c.recipe.egg part mentioned as the first one, but I got an "ImportError: Entry point ('zc.buildout', 'default') not found".
If you can help me dupe, I'm happy to look at it.
I haven't updated the bootstrap yet, so that might be the cause (though that'd be a bug).
If it is that you didn't update the bootstrap, then I stated it was one of the requirements. :-) Not inclined to call it a bug. Gary
On 08/27/2010 02:31 AM, Gary Poster wrote:
Only the script recipe seems to support it at the moment and it didn't look from the documentation that such a globally-found dependency would be valid for the entire buildout (and all the other parts). If I use that osc.recipe.sysegg and list that part first, it makes sure all other parts use the specified system eggs.
But then you are susceptible to many problems that occur in system Pythons that I've been debugging for this release.
Ok, you're trying to tell me not to do something, only I don't know what that "something" is :-) Do you mean it is a bad idea to use "matplotlib from the system packages" for the whole buildout? That it is better just to use it in the two or three parts where it is needed? It's not like I want my system matplotlib in one part and a locally failing-horribly buildout-installed matplotlib in another part. The current implementation makes it look like every single part that needs the system matplotlib needs to have a configuration for "look in the system for matplotlib". Ah! Did I perhaps misunderstand the 1.5.x changes? I'm used to have osc.recipe.sysegg pick two or three specific system eggs and add just those to the sys.path. Does the 1.5.x functionality only have a generic switch to switch on ALL system eggs? I'm getting lost :-)
Which django recipe do you use, by the way? I intend to update djangorecipe, using some changes that Attila made.
http://pypi.python.org/pypi/djangorecipe
Just a minute ago, I tried experimenting with a z3c.recipe.egg part mentioned as the first one, but I got an "ImportError: Entry point ('zc.buildout', 'default') not found".
If you can help me dupe, I'm happy to look at it.
I'll look in to it after next week. Two sites have a deadline then. So I'm happy with the 1.4.x-only bootstrap.py :-) Reinout -- Reinout van Rees - reinout@vanrees.org - http://reinout.vanrees.org Collega's gezocht! Django/python vacature in Utrecht: http://tinyurl.com/35v34f9
On Aug 27, 2010, at 4:23 AM, Reinout van Rees wrote:
On 08/27/2010 02:31 AM, Gary Poster wrote:
Only the script recipe seems to support it at the moment and it didn't look from the documentation that such a globally-found dependency would be valid for the entire buildout (and all the other parts). If I use that osc.recipe.sysegg and list that part first, it makes sure all other parts use the specified system eggs.
But then you are susceptible to many problems that occur in system Pythons that I've been debugging for this release.
Ok, you're trying to tell me not to do something, only I don't know what that "something" is :-)
:-) Fair enough. Here's my message. I realize that a system Python happened to work for you before. That's a bit awkward, because with pre-1.5.0 zc.buildout, using a system Python was not supported. Therefore, your recipe was not supported. Happily, zc.buildout does support a system Python. Moreover, it is designed to support your use case. Unhappily, your previously unsupported recipe does not work with it. Also unhappily, you can't switch to the 1.5 line until enough recipes have changed to make it possible for you to use it. For now, then, yay Buildout 1.4.4! Eventually, yay, you will be able to fulfill your recipe's use cases in a way that is much more robust against various things that can happen in a system Python!
Do you mean it is a bad idea to use "matplotlib from the system packages" for the whole buildout? That it is better just to use it in the two or three parts where it is needed?
It needs to be possible to do this.
It's not like I want my system matplotlib in one part and a locally failing-horribly buildout-installed matplotlib in another part.
No, but single buildouts are used to install different sections with entirely different requirements, including different Python versions.
The current implementation makes it look like every single part that needs the system matplotlib needs to have a configuration for "look in the system for matplotlib".
Yes, parts are individually installed, per my statement above. That said, I generally think it is a good idea for recipes to look in the buildout section for defaults to the pertinent options. That's what z3c.recipe.scripts does, and that's what I intend to do for the other recipes I upgrade. That means that, effectively, you would be able to make a single gesture in your buildout section and other parts will honor it.
Ah! Did I perhaps misunderstand the 1.5.x changes? I'm used to have osc.recipe.sysegg pick two or three specific system eggs and add just those to the sys.path. Does the 1.5.x functionality only have a generic switch to switch on ALL system eggs?
No. As mentioned before, the documentation for http://pypi.python.org/pypi/z3c.recipe.scripts should give you an idea on the features. Please see the options include-site-packages and allowed-eggs-from-site-packages in particular.
I'm getting lost :-)
Which django recipe do you use, by the way? I intend to update djangorecipe, using some changes that Attila made.
Cool, that's the one.
Just a minute ago, I tried experimenting with a z3c.recipe.egg part mentioned as the first one, but I got an "ImportError: Entry point ('zc.buildout', 'default') not found".
If you can help me dupe, I'm happy to look at it.
I'll look in to it after next week. Two sites have a deadline then. So I'm happy with the 1.4.x-only bootstrap.py :-)
Cool. :-) Gary
On 08/27/2010 04:12 PM, Gary Poster wrote:
I realize that a system Python happened to work for you before.
That's a bit awkward, because with pre-1.5.0 zc.buildout, using a system Python was not supported.
Therefore, your recipe was not supported.
"my recipe"? osc.recipe.sysegg is not something that *I* made, I'm just using it. "Is not supported" sounds like a helpdesk telling me "linux is not supported". And "not supported"? osc.recipe.sysegg installs the specific eggs I tell it to find (in the system python) as develop eggs. A nice trick that works pretty well.
No, but single buildouts are used to install different sections with entirely different requirements, including different Python versions.
The current implementation makes it look like every single part that needs the system matplotlib needs to have a configuration for "look in the system for matplotlib".
Yes, parts are individually installed, per my statement above.
What? I've never ever seen a buildout that has to work with different python versions in different parts. A plone buildout just uses one python. A django buildout, too. So it sounds to me like some impressive over-engineering? Note, I don't intend to offend with that statement :-) It's just that I'm thinking "I'll stick to 1.4.4 for the next half year and I'll re-check whether it all works again afterwards". And I don't like that kind of thought, being that I'm using buildout a lot. I introduced it in my current company. I (co-)wrote recipes. So it feels a bit dangerous if I'm not happily jumping on the newest version ;-) Anway... I'll delve deeper into it in one or two weeks :-) Reinout -- Reinout van Rees - reinout@vanrees.org - http://reinout.vanrees.org Collega's gezocht! Django/python vacature in Utrecht: http://tinyurl.com/35v34f9
On Aug 27, 2010, at 3:59 PM, Reinout van Rees wrote:
On 08/27/2010 04:12 PM, Gary Poster wrote:
I realize that a system Python happened to work for you before.
That's a bit awkward, because with pre-1.5.0 zc.buildout, using a system Python was not supported.
Therefore, your recipe was not supported.
"my recipe"? osc.recipe.sysegg is not something that *I* made, I'm just using it. "Is not supported" sounds like a helpdesk telling me "linux is not supported".
And "not supported"? osc.recipe.sysegg installs the specific eggs I tell it to find (in the system python) as develop eggs. A nice trick that works pretty well.
...Not sure what to tell you there. Jim has been clear from the beginning that system Pythons were not supported with buildout. I added support. Anything before then happened to work.
No, but single buildouts are used to install different sections with entirely different requirements, including different Python versions.
The current implementation makes it look like every single part that needs the system matplotlib needs to have a configuration for "look in the system for matplotlib".
Yes, parts are individually installed, per my statement above.
What? I've never ever seen a buildout that has to work with different python versions in different parts. A plone buildout just uses one python. A django buildout, too. So it sounds to me like some impressive over-engineering?
This is something that has been with buildout from the beginning, and has nothing to do with my changes. I've also found it useful, FWIW, both for applications and libraries.
Note, I don't intend to offend with that statement :-) It's just that I'm thinking "I'll stick to 1.4.4 for the next half year and I'll re-check whether it all works again afterwards". And I don't like that kind of thought, being that I'm using buildout a lot. I introduced it in my current company. I (co-)wrote recipes. So it feels a bit dangerous if I'm not happily jumping on the newest version ;-)
Anway... I'll delve deeper into it in one or two weeks :-)
OK. Gary
Gary Poster wrote:
And "not supported"? osc.recipe.sysegg installs the specific eggs I tell it to find (in the system python) as develop eggs. A nice trick that works pretty well.
...Not sure what to tell you there. Jim has been clear from the beginning that system Pythons were not supported with buildout.
This is somewhat puzzling. What makes a "system python" different from any other sort of python? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On Aug 30, 2010, at 1:07 PM, Chris Withers wrote:
Gary Poster wrote:
And "not supported"? osc.recipe.sysegg installs the specific eggs I tell it to find (in the system python) as develop eggs. A nice trick that works pretty well. ...Not sure what to tell you there. Jim has been clear from the beginning that system Pythons were not supported with buildout.
This is somewhat puzzling. What makes a "system python" different from any other sort of python?
Distributions install packages there. Gary
Gary Poster wrote:
On Aug 30, 2010, at 1:07 PM, Chris Withers wrote:
Gary Poster wrote:
And "not supported"? osc.recipe.sysegg installs the specific eggs I tell it to find (in the system python) as develop eggs. A nice trick that works pretty well. ...Not sure what to tell you there. Jim has been clear from the beginning that system Pythons were not supported with buildout. This is somewhat puzzling. What makes a "system python" different from any other sort of python?
Distributions install packages there.
Okay, but how is that any different from another source-compiled python in which someone has easy_installed packages? "system python" sounds wrong here, it feels like what's meant is "any python in which anything has been installed in site-packages", right? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On Mon, Aug 30, 2010 at 1:22 PM, Chris Withers <chris@simplistix.co.uk> wrote:
Gary Poster wrote:
On Aug 30, 2010, at 1:07 PM, Chris Withers wrote:
Gary Poster wrote:
And "not supported"? osc.recipe.sysegg installs the specific eggs I tell it to find (in the system python) as develop eggs. A nice trick that works pretty well.
...Not sure what to tell you there. Jim has been clear from the beginning that system Pythons were not supported with buildout.
This is somewhat puzzling. What makes a "system python" different from any other sort of python?
Distributions install packages there.
Okay, but how is that any different from another source-compiled python in which someone has easy_installed packages?
"system python" sounds wrong here, it feels like what's meant is "any python in which anything has been installed in site-packages", right?
It's not that "system python" is wrong, it just doesn't encompass all the things that are unsupported. You are correct in that a system Python is a "dirty" Python and dirty Python's may break buildout. -- Benji York
On Fri, Aug 27, 2010 at 4:23 PM, Gary Poster <gary.poster@canonical.com> wrote:
On Aug 27, 2010, at 3:59 PM, Reinout van Rees wrote:
On 08/27/2010 04:12 PM, Gary Poster wrote:
I realize that a system Python happened to work for you before.
That's a bit awkward, because with pre-1.5.0 zc.buildout, using a system Python was not supported.
Therefore, your recipe was not supported.
"my recipe"? osc.recipe.sysegg is not something that *I* made, I'm just using it. "Is not supported" sounds like a helpdesk telling me "linux is not supported".
And "not supported"? osc.recipe.sysegg installs the specific eggs I tell it to find (in the system python) as develop eggs. A nice trick that works pretty well.
...Not sure what to tell you there. Jim has been clear from the beginning that system Pythons were not supported with buildout. I added support. Anything before then happened to work.
I'm not sure I agree with this. :) Technically, you're right, in that I didn't offer any sympathy or support for people who had trouble using unclean Pythons (defined as Python installations that differ from a build from source). Still, I expected buildout to use anything found in the Python it was bootstrapped with, with all of the advantages and disadvantages that implied. :) This seemed the most conservative approach at the time and didn't cost me anything because I've never found it burdonsome to provide clean Python installes separate from the system Python. While, I think my decision was raesonable, I came to regret it. I prefer the approach taken by virtualenv to provide isolation by default and provide a way to relax it. My preferance is that buildout provide isolated and non-isolated modes and that the isolation apply equally to recipes, extensions, and eggs installed by parts. I'd be inclined to have buildout be non-islated by default, for backwards compatibility, and switch the default in buildout 2. I say all of this having done none of the recent work. :) Jim -- Jim Fulton
On Aug 30, 2010, at 1:50 PM, Jim Fulton wrote:
On Fri, Aug 27, 2010 at 4:23 PM, Gary Poster <gary.poster@canonical.com> wrote:
On Aug 27, 2010, at 3:59 PM, Reinout van Rees wrote:
On 08/27/2010 04:12 PM, Gary Poster wrote:
I realize that a system Python happened to work for you before.
That's a bit awkward, because with pre-1.5.0 zc.buildout, using a system Python was not supported.
Therefore, your recipe was not supported.
"my recipe"? osc.recipe.sysegg is not something that *I* made, I'm just using it. "Is not supported" sounds like a helpdesk telling me "linux is not supported".
And "not supported"? osc.recipe.sysegg installs the specific eggs I tell it to find (in the system python) as develop eggs. A nice trick that works pretty well.
...Not sure what to tell you there. Jim has been clear from the beginning that system Pythons were not supported with buildout. I added support. Anything before then happened to work.
I'm not sure I agree with this. :)
LOL. Sorry to misrepresent.
Technically, you're right, in that I didn't offer any sympathy or support for people who had trouble using unclean Pythons (defined as Python installations that differ from a build from source).
Right, this is what I was referring to.
Still, I expected buildout to use anything found in the Python it was bootstrapped with, with all of the advantages and disadvantages that implied. :)
OK.
This seemed the most conservative approach at the time and didn't cost me anything because I've never found it burdonsome to provide clean Python installes separate from the system Python.
While, I think my decision was raesonable, I came to regret it. I prefer the approach taken by virtualenv to provide isolation by default and provide a way to relax it.
My preferance is that buildout provide isolated and non-isolated modes and that the isolation apply equally to recipes, extensions, and eggs installed by parts.
Having fought through all this, I really think having separate isolation controls for the buildout (buildout, recipes, extensions) versus the things that are built by parts is the right way to go. It's two very different domains, and the people writing the two kinds of code are often completely separate.
I'd be inclined to have buildout be non-islated by default, for backwards compatibility, and switch the default in buildout 2.
As a meta comment, from the peanut gallery , I'm concerned about the buildout 2 plans (where "buildout 2" means "package name change") because they remind me of a couple of other similar historical changes that have permanently (or, at least, so far, often for years) divided communities. The forking is entirely your prerogative, of course. As to the practical issue of what to do now, here are options, working up to your stated preference. 1) Leave 1.5 and the trunk as it is, try to support osc.recipe.sysegg if someone directs me to the code, and move on, without a buildout 2. Unsurprisingly, that's my preference. (Post-weekend, I think I can see how to adapt the recipe Reinout listed to work with an isolated buildout, given the current way 1.5 works. I'd do the searching for eggs in a subprocess with the desired Python, and I suspect everything else would work out fine, with the added robustness of 1.5.0's general isolation. Since the recipe has no tests in the tarball and Google doesn't show me the sourcecode, I'm not yet encouraged to give it a whirl, but could be.) 2) Built eggs are non-isolated by default if you use zc.recipe.egg. Switching to z3c.recipe.scripts is an explicit gesture. However, I'm adjusting *other* recipes to use code from z3c.recipe.scripts to generate their scripts, and these (zc.recipe.testrunner and z3c.recipe.tag for now, z3c.recipe.i18n locally on my machine, djangorecipe intended to come possibly among others) are defaulting to isolation. I could make these other recipes default to not isolating. I'd regard that as a shame, because I think it is really nice for buildout to be able to be used with a system Python easily without having to know about special switches or anything else, but practically it would be pretty easy for me. 3) It would be more complex but doable to make all buildout bits (recipes, extensions, and buildout itself) default to not being isolated. Given the current architecture, which I have no desire to change myself, I think it would have to involve one bootstrap for an isolated environment and one bootstrap for a non-isolated environment, and then various changes within Buildout itself. I don't like this, but it is what I see would be needed to support the current version of osc.recipe.sysegg. 4) We can have unified isolation controls for buildout (+ recipes and extensions) and the built eggs. FWIW, I'm happy with option 1, and helping out with osc.recipe.sysegg if given some direction. I'm OK with option 2, and doing the work. I'm unhappy with option 3, but would try to do the work. I'd have to be further convinced about option 4. Gary
On 08/30/2010 09:06 PM, Gary Poster wrote:
(Post-weekend, I think I can see how to adapt the recipe Reinout listed to work with an isolated buildout, given the current way 1.5 works. I'd do the searching for eggs in a subprocess with the desired Python, and I suspect everything else would work out fine, with the added robustness of 1.5.0's general isolation. Since the recipe has no tests in the tarball and Google doesn't show me the sourcecode, I'm not yet encouraged to give it a whirl, but could be.)
I mailed the author for something unrelated to this discussion about two weeks ago. I didn't hear anything back, yet. - Author doesn't reply quickly (ok, it could be vacation time :-) ) - No findable source code. => Don't put (too much) effort into this. Rewriting the recipe by doing the thing you suggest (subprocess with the python buildout was started with) sounds doable. Just install a couple of explicitly searched-for packages from that system python as development eggs or so. That would give me what I want: "just use the installed debian package for matplotlib and mapnik" for all parts in the buildout. The subprocess hack sounds a bit dirty, though. It makes me feel like I'm working around buildout. Reinout -- Reinout van Rees - reinout@vanrees.org - http://reinout.vanrees.org Collega's gezocht! Django/python vacature in Utrecht: http://tinyurl.com/35v34f9
On Aug 30, 2010, at 4:16 PM, Reinout van Rees wrote:
On 08/30/2010 09:06 PM, Gary Poster wrote:
(Post-weekend, I think I can see how to adapt the recipe Reinout listed to work with an isolated buildout, given the current way 1.5 works. I'd do the searching for eggs in a subprocess with the desired Python, and I suspect everything else would work out fine, with the added robustness of 1.5.0's general isolation. Since the recipe has no tests in the tarball and Google doesn't show me the sourcecode, I'm not yet encouraged to give it a whirl, but could be.)
I mailed the author for something unrelated to this discussion about two weeks ago. I didn't hear anything back, yet.
- Author doesn't reply quickly (ok, it could be vacation time :-) )
- No findable source code.
=> Don't put (too much) effort into this.
OK, thank you for heads up.
Rewriting the recipe by doing the thing you suggest (subprocess with the python buildout was started with) sounds doable. Just install a couple of explicitly searched-for packages from that system python as development eggs or so.
That would give me what I want: "just use the installed debian package for matplotlib and mapnik" for all parts in the buildout.
The subprocess hack sounds a bit dirty, though. It makes me feel like I'm working around buildout.
FWIW, It's what buildout does for itself all over the place. It's also reasonable to my mind if you want to explicitly break through an isolation barrier like this, but I won't argue taste. Gary
On Mon, Aug 30, 2010 at 4:47 PM, Gary Poster <gary.poster@canonical.com> wrote:
On Aug 30, 2010, at 4:16 PM, Reinout van Rees wrote:
...
The subprocess hack sounds a bit dirty, though. It makes me feel like I'm working around buildout.
FWIW, It's what buildout does for itself all over the place. It's also reasonable to my mind if you want to explicitly break through an isolation barrier like this, but I won't argue taste.
Buildout uses subprocess to run setup scripts. That's pretty much all. It runs setups scripts in a subprocess because they are written expecting to be the main program and could do all sorts of things that could break buildout if run in process. Jim -- Jim Fulton
On Mon, Aug 30, 2010 at 3:06 PM, Gary Poster <gary.poster@canonical.com> wrote:
On Aug 30, 2010, at 1:50 PM, Jim Fulton wrote:
...
My preferance is that buildout provide isolated and non-isolated modes and that the isolation apply equally to recipes, extensions, and eggs installed by parts.
Having fought through all this, I really think having separate isolation controls for the buildout (buildout, recipes, extensions) versus the things that are built by parts is the right way to go. It's two very different domains, and the people writing the two kinds of code are often completely separate.
As a meta comment, from the peanut gallery , I'm concerned about the buildout 2 plans (where "buildout 2" means "package name change") because they remind me of a couple of other similar historical changes that have permanently (or, at least, so far, often for years) divided communities. The forking is entirely your
I don't want to make work for you, but I don't understand what issues you fought. Are there any links you can give that might allow me to get this for myself? prerogative, of course. I wasn't planning on a fork, a name change, or to do anything disruptive. My main desire would be to try to see if I can simplify the implementation, *especially* wrt tests. The reason for the "2" would be mainly to change some defaults in non-backward-compatable ways, including: - Isolated by default - Prefer final by default - unzip by default
As to the practical issue of what to do now, here are options, working up to your stated preference.
...
2) Built eggs are non-isolated by default if you use zc.recipe.egg.
I don't know what you mean by "built eggs". ... Jim -- Jim Fulton
On Mon, Aug 30, 2010 at 05:17:41PM -0400, Jim Fulton wrote:
I wasn't planning on a fork, a name change, or to do anything disruptive.
My main desire would be to try to see if I can simplify the implementation, *especially* wrt tests. The reason for the "2" would be mainly to change some defaults in non-backward-compatable ways, including: ... - unzip by default
YEEEEEEEEES!!!! Sorry, got carried away there for a bit. Marius Gedminas -- MSDOS didn't get as bad as it is overnight -- it took over ten years of careful development. -- dmeggins@aix1.uottawa.ca
On Aug 30, 2010, at 5:17 PM, Jim Fulton wrote:
On Mon, Aug 30, 2010 at 3:06 PM, Gary Poster <gary.poster@canonical.com> wrote:
On Aug 30, 2010, at 1:50 PM, Jim Fulton wrote:
...
My preferance is that buildout provide isolated and non-isolated modes and that the isolation apply equally to recipes, extensions, and eggs installed by parts.
I have a branch that I've asked Reinout to test that does this. Defaulting to non-isolated should significantly increase backwards compatibility. I'll hope to release a 1.5.2 with this next week. It will also have some other bugfixes found along the way, and maybe some Mac support (turns out the Mac system Python does something weird and different itself).
Having fought through all this, I really think having separate isolation controls for the buildout (buildout, recipes, extensions) versus the things that are built by parts is the right way to go. It's two very different domains, and the people writing the two kinds of code are often completely separate.
I don't want to make work for you, but I don't understand what issues you fought. Are there any links you can give that might allow me to get this for myself?
I talked to Jim after this email, and my answer was essentially, "No. Maybe I was just tired fighting generally and gave up too easily." Gary
Gary Poster wrote:
It's not like I want my system matplotlib in one part and a locally failing-horribly buildout-installed matplotlib in another part.
No, but single buildouts are used to install different sections with entirely different requirements, including different Python versions.
This feels like needless complexity to me. If a different python is needed, it should be in its own buildout. If you need to bundle a bunch of buildouts together because of this, use a recipe that runs "sub buildouts" in a separate process... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On Aug 30, 2010, at 1:06 PM, Chris Withers wrote:
Gary Poster wrote:
It's not like I want my system matplotlib in one part and a locally failing-horribly buildout-installed matplotlib in another part. No, but single buildouts are used to install different sections with entirely different requirements, including different Python versions.
This feels like needless complexity to me.
If a different python is needed, it should be in its own buildout. If you need to bundle a bunch of buildouts together because of this, use a recipe that runs "sub buildouts" in a separate process...
- In my experience, that would have been more annoying than the feature that exists. - What you describe does not exist. - It's from before my time; you can argue it with Jim if you like. I've liked it and used it. Gary
On Mon, Aug 30, 2010 at 1:06 PM, Chris Withers <chris@simplistix.co.uk> wrote:
Gary Poster wrote:
It's not like I want my system matplotlib in one part and a locally failing-horribly buildout-installed matplotlib in another part.
No, but single buildouts are used to install different sections with entirely different requirements, including different Python versions.
This feels like needless complexity to me.
This is a requirement we had here at ZC. It wasn't needless.
If a different python is needed, it should be in its own buildout. If you need to bundle a bunch of buildouts together because of this, use a recipe that runs "sub buildouts" in a separate process...
It's possible that this would be a better approach. It's true that supporting multiple Python interpreters is a pain. I don't have this need atm, so I'm not motivated to try your solution. :) I wonder what other people think. Does anyone else have current need to deal with multiple Python versions in the same buildout? Jim -- Jim Fulton
On Aug 30, 2010, at 1:27 PM, Jim Fulton wrote:
On Mon, Aug 30, 2010 at 1:06 PM, Chris Withers <chris@simplistix.co.uk> wrote:
Gary Poster wrote:
It's not like I want my system matplotlib in one part and a locally failing-horribly buildout-installed matplotlib in another part.
No, but single buildouts are used to install different sections with entirely different requirements, including different Python versions.
This feels like needless complexity to me.
This is a requirement we had here at ZC. It wasn't needless.
If a different python is needed, it should be in its own buildout. If you need to bundle a bunch of buildouts together because of this, use a recipe that runs "sub buildouts" in a separate process...
It's possible that this would be a better approach. It's true that supporting multiple Python interpreters is a pain. I don't have this need atm, so I'm not motivated to try your solution. :)
I wonder what other people think. Does anyone else have current need to deal with multiple Python versions in the same buildout?
Launchpad has needed to when we needed to generate scripts that were only supported by Python version X but Launchpad only supported version Y. This was typically in transition periods, so the feature was very nice--we would be switching back to a uniform Python soon, but meanwhile, the buildout and its components still worked. That kind of thing is generally my most compelling usage. I've also used it (or seen it used? I forget) to set up multiple test runners for different Python versions, for library packages. That's convenient sometimes. I could imagine these being done in a way as Chris described it, but to be convenient I'd want to share configuration across the buildouts--and meanwhile, the feature that does exist has suited nicely. Gary
On Mon, Aug 30, 2010 at 2:10 PM, Gary Poster <gary.poster@canonical.com> wrote:
On Aug 30, 2010, at 1:27 PM, Jim Fulton wrote:
On Mon, Aug 30, 2010 at 1:06 PM, Chris Withers <chris@simplistix.co.uk> wrote:
Gary Poster wrote:
It's not like I want my system matplotlib in one part and a locally failing-horribly buildout-installed matplotlib in another part.
No, but single buildouts are used to install different sections with entirely different requirements, including different Python versions.
This feels like needless complexity to me.
This is a requirement we had here at ZC. It wasn't needless.
If a different python is needed, it should be in its own buildout. If you need to bundle a bunch of buildouts together because of this, use a recipe that runs "sub buildouts" in a separate process...
It's possible that this would be a better approach. It's true that supporting multiple Python interpreters is a pain. I don't have this need atm, so I'm not motivated to try your solution. :)
I wonder what other people think. Does anyone else have current need to deal with multiple Python versions in the same buildout?
Launchpad has needed to when we needed to generate scripts that were only supported by Python version X but Launchpad only supported version Y. This was typically in transition periods, so the feature was very nice--we would be switching back to a uniform Python soon, but meanwhile, the buildout and its components still worked.
That kind of thing is generally my most compelling usage.
I've also used it (or seen it used? I forget) to set up multiple test runners for different Python versions, for library packages. That's convenient sometimes.
Now that you mention it, I've used it this way too recently, however recent turmoil with the Zope test runner make this use case break for me. Some subtleties arise from the fact that you can only use one Python for the recipes and the recipes will sometimes depend in significant ways on the Python used.
I could imagine these being done in a way as Chris described it, but to be convenient I'd want to share configuration across the buildouts
This could be done using a files that were extended. It might even be practical to share a single config and pass overrides to the sub-buildout. It's hard to know without trying it. :)
--and meanwhile, the feature that does exist has suited nicely.
True, however, It has provided faitly painful to maintain, especially wrt tests. Jim -- Jim Fulton
On 08/30/2010 07:27 PM, Jim Fulton wrote:
I wonder what other people think. Does anyone else have current need to deal with multiple Python versions in the same buildout?
I was surprised it was even possible :-) Never heard of the option till a couple of days ago. I've used buildout for 4 years or so and I've never had the need to use multiple pythons. Everything was just a single plone/grok/django project with some scripts and recipes. Reinout -- Reinout van Rees - reinout@vanrees.org - http://reinout.vanrees.org Collega's gezocht! Django/python vacature in Utrecht: http://tinyurl.com/35v34f9
On Mon, Aug 30, 2010 at 22:18, Reinout van Rees <reinout@vanrees.org> wrote:
I've used buildout for 4 years or so and I've never had the need to use multiple pythons. Everything was just a single plone/grok/django project with some scripts and recipes.
"me too", and I'm a heavy user of buildout (we use it for dev and prod). -- Sebastien Douche <sdouche@gmail.com> Twitter: http://bit.ly/afkrK (agile, lean, python, open source)
Le 30/08/10 19:27, Jim Fulton a écrit :
On Mon, Aug 30, 2010 at 1:06 PM, Chris Withers<chris@simplistix.co.uk> wrote:
Gary Poster wrote:
It's not like I want my system matplotlib in one part and a locally failing-horribly buildout-installed matplotlib in another part.
No, but single buildouts are used to install different sections with entirely different requirements, including different Python versions.
This feels like needless complexity to me.
This is a requirement we had here at ZC. It wasn't needless.
If a different python is needed, it should be in its own buildout. If you need to bundle a bunch of buildouts together because of this, use a recipe that runs "sub buildouts" in a separate process...
It's possible that this would be a better approach. It's true that supporting multiple Python interpreters is a pain. I don't have this need atm, so I'm not motivated to try your solution. :)
I wonder what other people think. Does anyone else have current need to deal with multiple Python versions in the same buildout?
Jim
I am developing vimpdb which I obviously want to be compatible with multiple python versions. I was planning to write a single buildout to build the various testrunners. -- Godefroid Chapelle (aka __gotcha) http://bubblenet.be
On Thu, Sep 2, 2010 at 6:39 AM, Godefroid Chapelle <gotcha@bubblenet.be> wrote:
Le 30/08/10 19:27, Jim Fulton a écrit :
On Mon, Aug 30, 2010 at 1:06 PM, Chris Withers<chris@simplistix.co.uk> wrote:
Gary Poster wrote:
It's not like I want my system matplotlib in one part and a locally failing-horribly buildout-installed matplotlib in another part.
No, but single buildouts are used to install different sections with entirely different requirements, including different Python versions.
This feels like needless complexity to me.
This is a requirement we had here at ZC. It wasn't needless.
If a different python is needed, it should be in its own buildout. If you need to bundle a bunch of buildouts together because of this, use a recipe that runs "sub buildouts" in a separate process...
It's possible that this would be a better approach. It's true that supporting multiple Python interpreters is a pain. I don't have this need atm, so I'm not motivated to try your solution. :)
I wonder what other people think. Does anyone else have current need to deal with multiple Python versions in the same buildout?
Jim
I am developing vimpdb which I obviously want to be compatible with multiple python versions.
I was planning to write a single buildout to build the various testrunners.
I encourage you to try it and see how it works for you, and report back. I've done this in the past, but: - It was awkward for other developers who didn't have the same versions of Python installed and who didn't generally have the mind set to try to actively develop against multiple versions simultaneously. - The fact that the recipes all run in the buildout process, and thus with whatever version of buildout was used, led to odd errors. For example the zc.testrunner recipe had (maybe still has) trouble generating test scripts for target Python versions different than what it was run with. (It didn't always have this problem.) Jim -- Jim Fulton
Le 02/09/10 14:53, Jim Fulton a écrit :
On Thu, Sep 2, 2010 at 6:39 AM, Godefroid Chapelle<gotcha@bubblenet.be> wrote:
I am developing vimpdb which I obviously want to be compatible with multiple python versions.
I was planning to write a single buildout to build the various testrunners.
I encourage you to try it and see how it works for you, and report back.
I will.
I've done this in the past, but:
- It was awkward for other developers who didn't have the same versions of Python installed and who didn't generally have the mind set to try to actively develop against multiple versions simultaneously.
- The fact that the recipes all run in the buildout process, and thus with whatever version of buildout was used, led to odd errors. For example the zc.testrunner recipe had (maybe still has) trouble generating test scripts for target Python versions different than what it was run with. (It didn't always have this problem.)
Thanks for that information
Jim
-- Godefroid Chapelle (aka __gotcha) http://bubblenet.be
Maurits van Rees wrote:
What I *do* see is that when you start with zc.buildout 1.5.0, then change the pin to 1.4.3 and rerun the buildout, this fails when you are using a buildout extension. The other way around (first 1.4.3, then change it to 1.5.0) works fine. I tried it with buildout-versions and with mr.developer as extensions, both having the same result. For example, start with this, and any bootstrap.py:
If it turns out that buildout-versions is doing anything "wrong", please do let me know and I'll see what I can do to fix it! cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
participants (11)
-
Attila Oláh -
Barry Warsaw -
Benji York -
Chris Withers -
Gary Poster -
Godefroid Chapelle -
Jim Fulton -
Marius Gedminas -
Maurits van Rees -
Reinout van Rees -
Sebastien Douche