On Apr 30, 2010, at 11:16 AM, Jonathan Ballet wrote:

Hi,

On Thu, 29 Apr 2010 22:28:33 -0400, Gary Poster <gary.poster@canonical.com> wrote:

...

I have made a beta release of zc.buildout 1.5.0.

Among other changes, this release includes the ability to use zc.buildout with a system Python, if you use the new z3c.recipe.scripts instead of zc.recipe.egg for generating scripts.

Changelog:

http://pypi.python.org/pypi/zc.buildout/1.5.0b1#b1-2010-04-29

Additional comments, with respects to the mail of Chris McDonough (I hit the same kind of issues).

1. It looks like something has changed in the way bootstrap run, in relation to the "buildout:index" value in buildout.cfg. Here, we use an internal Pypi index, which points to a list of *our* packages, and so, the value of "buildout:index" refers to this server. However, it seems now that bootstrap take this value to look for setuptools/distribute packages while bootstrapping. In our configuration, this fails with the following error:

$ python bootstrap.py Creating directory '/home/jon/projects/sa/repos/unstable/sact.dbapi/bin'. Creating directory '/home/jon/projects/sa/repos/unstable/sact.dbapi/parts'. Creating directory '/home/jon/projects/sa/repos/unstable/sact.dbapi/eggs'. Creating directory '/home/jon/projects/sa/repos/unstable/sact.dbapi/develop-eggs'. Couldn't find index page for 'distribute' (maybe misspelled?) Getting distribution for 'distribute'. While: Bootstrapping. Getting distribution for 'distribute'. Error: Couldn't find a distribution for 'distribute'.

.. since our server doesn't contain this package. If I create a ~/.pydistutils.cfg, with "easy_install:index-url" value to the official Pypi, everything goes fine (bootstrap + buildout).

Yes, this is the result of the new feature listed in the changes file for bootstrap.py: "The buildout script generated by bootstrap honors more of the settings in the designated configuration file (e.g., buildout.cfg)." This matched our desire: for instance, if we want to use our own index to manage the packages needed to build out software , that should include zc.buildout. When an earlier version of bootstrap did not honor the config file, I was surprised, so I felt this was an improvement, and perhaps even arguably a bugfix. I'd like to hear input on whether this change should be reverted. I prefer it, and for the particular instance you describe I'd suggest that you keep zc.buildout and your other dependencies on your own server if you already have that infrastructure, but...I won't push back too hard if folks disagree.

2. In the previous point, I omitted to say that we are using an old bootstrap.py, a little bit customized (with a dirty hack which gets ez_setup.py on our internal server (sounds familiar? ;))

heh :-)

I tried to use the latest bootstrap.py, and I ran into the same problem as Chris, since my Python executable was from a Virtualenv.

I dig a little bit into the problem, and it looks like the -S option bootstrap.py passes to Python executable if it detects that it runs from a non-isolated environment causes this problem. In fact, Virtualenv currently only copies (with symlinks) some of the stdlib modules into its environment (I'm not sure why specifically those), and also add the global stdlib directory into sys.path. However, running Virtualenv's Python with the -S remove the global stdlib from the sys.path! (in particular, it removes /usr/lib/python2.5 on my machine):

I'm not sure this could help, but this was just a sharing of what I found.

Yes, thank you. As I confirmed in another thread, this is exactly the problem. I didn't understand the importance of what you wrote until I saw it myself.

Despite those problems, I'm eager to test further this new release of Buildout, thanks for your work!

Thank you for your feedback. Gary

Hi Lennart. I cannot duplicate this when running the tests for zc.recipe.testrunner, which was what you appeared to be doing. I do see some test failures in that package, some of them caused by the buildout change, but they are reasonable: the tests check for the actual script output, which is fragile, and which broke. I used Python 2.6, as you appeared to be. I have tried running with both 1.5.0b1 and a version of my current betafix branch. Running within a --no-site-packages virtualenv and b1 did break with errors like "IOError: [Errno 2] No such file or directory: '/tmp/tmpHXetXtbuildoutSetUp/_TEST_/sample-buildout/parts/buildout/orig-prefix.txt'". That doesn't look like the error you are reporting. These also are fixed with my betafix branch. Can you give me some instructions to duplicate? Thanks Gary On Apr 30, 2010, at 9:46 PM, Lennart Regebro wrote:

Another bug:

In code like this:

def setUp(test): zc.buildout.testing.buildoutSetUp(test) zc.buildout.testing.install_develop('zc.recipe.testrunner', test) zc.buildout.testing.install_develop('zc.recipe.egg', test) zc.buildout.testing.install('zope.testing', test) zc.buildout.testing.install('zope.testrunner', test) zc.buildout.testing.install('zope.interface', test) zc.buildout.testing.install('zope.exceptions', test)

You get the following error:

Traceback (most recent call last): File "/opt/python26/lib/python2.6/doctest.py", line 2120, in setUp self._dt_setUp(test) File "/home/projects/zc.recipe.testrunner/trunk/src/zc/recipe/testrunner/tests.py", line 25, in setUp zc.buildout.testing.buildoutSetUp(test) File "/home/projects/zc.recipe.testrunner/trunk/zc.buildout-1.5.0b1-py2.6.egg/zc/buildout/testing.py", line 332, in buildoutSetUp make_buildout() File "/home/projects/zc.recipe.testrunner/trunk/zc.buildout-1.5.0b1-py2.6.egg/zc/buildout/testing.py", line 272, in make_buildout ).bootstrap([]) File "/home/projects/zc.recipe.testrunner/trunk/zc.buildout-1.5.0b1-py2.6.egg/zc/buildout/buildout.py", line 373, in bootstrap include_site_packages=False, File "/home/projects/zc.recipe.testrunner/trunk/zc.buildout-1.5.0b1-py2.6.egg/zc/buildout/easy_install.py", line 1030, in install return installer.install(specs, working_set) File "/home/projects/zc.recipe.testrunner/trunk/zc.buildout-1.5.0b1-py2.6.egg/zc/buildout/easy_install.py", line 843, in install for dist in self._get_dist(requirement, ws, self._always_unzip): File "/home/projects/zc.recipe.testrunner/trunk/zc.buildout-1.5.0b1-py2.6.egg/zc/buildout/easy_install.py", line 678, in _get_dist raise MissingDistribution(requirement, ws) MissingDistribution: Couldn't find a distribution for 'distribute'.

Going back to 1.4.3 (and zc.recipe.egg 1.2.2 solves the problem.

-- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64

On Jul 22, 2010, at 12:47 PM, Lennart Regebro wrote:

On Thu, Jul 15, 2010 at 15:15, Gary Poster <gary.poster@canonical.com> wrote:

...

Hi Lennart.

I cannot duplicate this when running the tests for zc.recipe.testrunner, which was what you appeared to be doing.

Well, in that case it probably works now. I'll try again (with the betafix branch, right?)

Yeah, exactly (though like I said, I couldn't dupe it with 1.5.0b1 either) Thanks Gary

On Jul 23, 2010, at 6:46 AM, Lennart Regebro wrote:

On Thu, Jul 22, 2010 at 19:10, Gary Poster <gary.poster@canonical.com> wrote:

...

...

Well, in that case it probably works now. I'll try again (with the betafix branch, right?)

Yeah, exactly (though like I said, I couldn't dupe it with 1.5.0b1 either)

OK, the way to duplicate it is to run the tests with

$python setup.py test

To make that happen with 1.5.0dev, you have to make an sdist, put that in the trunk directory, and also add

dependency_links = ['.'],

to setup.py parameters.

The error doesn't appear if you run bin/test.

Ah, ok. To my knowledge, these packages don't support that spelling. bin/test is the way to run the tests for the Zope packages AFAIK. Do you, or others, disagree? The fact that you have to modify setup.py in order to run your example seems to support my perspective. If you disagree, I'll see if I can address, but I'd rather not add that to the release blockers if I don't have to.

Although I get other errors when I do that. However, they are not related to your branch, that happens with 1.5.0b2 as well:

Failed example: print system(os.path.join(sample_buildout, 'bin', 'buildout') + ' -q'), Expected nothing Got: install_dir /tmp/tmpa_M-fRbuildoutSetUp/_TEST_/sample-buildout/develop-eggs/tmpDfJdS0build install_dir /tmp/tmpa_M-fRbuildoutSetUp/_TEST_/sample-buildout/develop-eggs/tmpDfNtGFbuild install_dir /tmp/tmpa_M-fRbuildoutSetUp/_TEST_/sample-buildout/develop-eggs/tmpuSWFFvbuild

The install_dir stuff appears to be something that distribute adds. They didn't show up with setuptools. I just made the buildout tests in my betafix branch ignore them. As far as I know, that's a reasonable approach. Gary

On Jul 23, 2010, at 10:07 AM, Lennart Regebro wrote:

On Fri, Jul 23, 2010 at 14:05, Gary Poster <gary.poster@canonical.com> wrote:

...

To my knowledge, these packages don't support that spelling. bin/test is the way to run the tests for the Zope packages AFAIK. Do you, or others, disagree? The fact that you have to modify setup.py in order to run your example seems to support my perspective.

I only need to modify it to run it with unreleased modules.

...

If you disagree, I'll see if I can address, but I'd rather not add that to the release blockers if I don't have to.

I disagree,

OK, I'll look into it. Gary

On Apr 30, 2010, at 4:38 PM, Tarek Ziadé wrote:

On Fri, Apr 30, 2010 at 4:28 AM, Gary Poster <gary.poster@canonical.com> wrote:

...

I have made a beta release of zc.buildout 1.5.0.

Among other changes, this release includes the ability to use zc.buildout with a system Python, if you use the new z3c.recipe.scripts instead of zc.recipe.egg for generating scripts.

Changelog:

http://pypi.python.org/pypi/zc.buildout/1.5.0b1#b1-2010-04-29

In my experience, working with a system ("non-clean") Python can be fragile in many ways. However, this release addresses the problems I have encountered so far.

Your feedback is welcome. I hope to make a final release sometime next week.

There's a big problem in our ecosystem: PyPI + our various installers don't have the concept of "beta" release. Any new release is considered to be the latest release.

Until it's fixed, I think we should avoid pushing any betas to PyPI, especially zc.buildout, because many buildout instances out there are getting upgraded automatically. Leading to a wave of problems when there's an issue in the release.

I propose that we "hide" the beta version on PyPI until all problems are cleared u, so people don't auto-upgrade their buildouts.

I have done so, for zc.buildout and zc.recipe.egg. However, merely hiding them is apparently insufficient, in my tests. I am investigating other options.

Moreover, zc.buildout is very fragile to all the different setups out there. We have been doing manual tests in the past, but we should definitely set up some continuous integration tools. We should try, if anyone has some time, to implement a few functional tests where buildout would be used in various environments.

- with/without virtualenv), - a typical plone setup - etc.

Last but not least, I think this auto-upgrade feature zc.buildout should be removed. I'd be in favor of an explicit update of this software, rather than having zc.buildout auto-upgrading itself like this.

Having a system broken because a new version was pushed at PyPI is not something we should end up with.

I'm in favor of this general statement, certainly.

The auto upgrade feature should be optional (and disabled by default IMHO, or at least asking the question at the prompt ?).

Regards Tarek

...

Gary _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

-- Tarek Ziadé | http://ziade.org

On Sat, May 1, 2010 at 12:01 AM, Gary Poster <gary.poster@canonical.com> wrote:

I have re-released zc.buildout 1.4.3 as 1.5.0b2, and re-released zc.recipe.egg 1.2.2 as 1.2.3b2.  My tests seem to show everything back to normal--I was able to build Chris's initial problem example as a smoketest.

Next week I'll announce betas *not* uploaded to PyPI.  These will solve the problems identified today.  I'll provide instructions on how to download and install them locally for tests.

I appreciate your feedback today, and will appreciate it even more when you try out those betas next week!

In closing, I'll summarize the problems identified today with 1.5.0b1, with their statuses.

- virtualenv + zc.buildout fails.  1.5.0b1 expected the -S of executables to work properly, and it doesn't for virtualenv.  This may not be fixable within virtualenv, per Carl's email.  I have a change to zc.buildout that I think will work around this in a local branch.  Tests fail, so it needs more work.  This is the most critical problem identified.

- zc.buildout's bootstrap honors more options in the .cfg file, which caused Jonathan some grief.  I'm classifying this as a "Won't Fix" unless I get pushback, per my earlier mail.

- If you have buildout installed in your system Python's site-packages, bootstrap falls over.  I have not duped this but I think I know the cause.  I'll try to dupe, and ask Rodrigo to confirm a fix when I'm ready next week.

- zc.recipe.testing's tests showed some API calls to zc.buildout that failed mysteriously.  I have not investigated that yet.  I know of several other existing recipes that use zc.buildout's API that work fine.  I'm hopeful that this will be easy to diagnose and address.

- Tarek raised the problem that we can't easily handle beta packages of critical packaging infrastructure.  Jim countered with information on how buildout already does handle it.  However, I don't think that this helps with bootstrapping.  Moreover, it requires explicit configuration to get some of the behavior Jim describes.  I plan to address Tarek's concerns for now by having betas available through alternate means than PyPI.  However, this will require more active participation from you all to help prepare a smooth release on PyPI.

Thanks for your work ! for "beta" releases, you could use the "dev" tag, and leave the code in the svn trunk. If the metadata of zc.buildout or properly set, we can install the trunk like this: $ easy_install zc.buildout == dev in order to make this work, we need to add somewhere in the documentation (long_description) a link to the svn trunk, followed by an egg fragment marker, and release the metadata with it: http://svn.zope.org/repos/main/zc.buildout/trunk#egg=zc.buildout-dev This will be scanned by easy_install and recognized as the dev version. Pip is not compatible with this feature IIRC. The bootstrap script should also work with this. People can force the dev version in their buildout config as well, if the latest metadata at PyPI has this dev marker. This will make it easier for you to ask people to try the trunk before you release it, Regards Tarek

Thank you all for your help.  Talk to you next week.

Gary _______________________________________________ Distutils-SIG maillist  -  Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

-- Tarek Ziadé | http://ziade.org

On Fri, Apr 30, 2010 at 11:02 PM, Gary Poster <gary.poster@canonical.com> wrote: [..]

...

I propose that we "hide" the beta version on PyPI until all problems are cleared u, so people don't auto-upgrade their buildouts.

I have done so, for zc.buildout and zc.recipe.egg.  However, merely hiding them is apparently insufficient, in my tests.

Ooops, I thought hidden release weren't showing up in the index

On Fri, Apr 30, 2010 at 11:17 PM, Jim Fulton <jim@zope.com> wrote: [..]

...

There's a big problem in our ecosystem:  PyPI + our various installers don't have the concept of "beta" release. Any new release is considered to be the latest release.

buildout does. :)

buildout has a prefer-final option.

Including:

 [buildout]  prefer-final = true

Will cause buildout to only use final release unless there are no final releases that satisfy a requirement.

Using this option will cause buildout to automatically downgrade itself if it previously upgraded to a non-final release. In buildout 2, prefer-final true will be the default.

Ah, great ! on PyPI side, we will be able to mark releases as being beta thanks to PEP 386. Maybe the simple index could differentiate them somehow. [..]

...

Last but not least, I think this auto-upgrade feature zc.buildout should be removed.  I'd be in favor of an explicit update of this software, rather than having zc.buildout auto-upgrading itself like this.

I find auto update to be useful, but I wouldn't object to a manual command:

 bin/buildout upgrade

It would be easier to implement than what we have now.

+1. An explicit upgrade step would be perfect: people won't have to prevent auto-upgrades by adding options. I am wondering how hard it would be to apply the same principal to all eggs in the buildout, but maybe this is a bit out of scope Tarek -- Tarek Ziadé | http://ziade.org

On Sat, May 1, 2010 at 4:35 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote: ...

An explicit upgrade step would be perfect: people won't have to prevent auto-upgrades by adding options. I am wondering how hard it would be to apply the same principal to all eggs in the buildout, but maybe this is a bit out of scope

It's easy. Run buildout with -N or use: [buildout] newest = False And, of course, buildout also let's you specify specific versions for projects, via either project requirements or a versions section. Jim -- Jim Fulton

On Fri, Apr 30, 2010 at 23:17, Jim Fulton <jim@zope.com> wrote:

...

There's a big problem in our ecosystem:  PyPI + our various installers don't have the concept of "beta" release. Any new release is considered to be the latest release.

buildout does. :)

buildout has a prefer-final option.

I think it's another need (buildout vs libs). Another point, with this feature activated, some eggs aren't downloadable (thanks to the stupid name chosen by the developers) -- Sebastien Douche <sdouche@gmail.com> Twitter: http://bit.ly/afkrK (agile, lean, python, open source)

On Mon, May 3, 2010 at 15:58, Gary Poster <gary.poster@canonical.com> wrote:

Alternatively, try cleaning out the buildout stuff (this usually works for me: ``rm -rf .installed.cfg bin parts``) and then rerun bootstrap and bin/buildout.

Yep, that helped. Thanks for helping out. And sorry for making noise. Attila