[buildout] windows/jenkins annoyance
Hi Jim, This is an example of the Windows problem I was referring to: http://jenkins.simplistix.co.uk/job/testfixtures-buildout/160/PYTHON=2.5,lab... C:\Python25\python.exe bootstrap.py ...appears to fork off a process in such a way that it returns instantly and then runs in the background. Of course, Jenkins has no way to know how long to wait for, so problems such as those above occur. My horrible hack around this is: https://github.com/Simplistix/testfixtures/blob/master/jenkins.bat#L2 ...which waits 20 seconds and then moves to the next step. It's far from reliable, especially when the Windows box has any load on it. Can you make any suggestions how to solve this problem? cheers, Chris PS: Where does the buildout continuous integration testing live nowadays? -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On Sat, Jan 19, 2013 at 3:42 PM, Chris Withers <chris@simplistix.co.uk> wrote:
Hi Jim,
This is an example of the Windows problem I was referring to:
http://jenkins.simplistix.co.uk/job/testfixtures-buildout/160/PYTHON=2.5,lab...
C:\Python25\python.exe bootstrap.py
...appears to fork off a process in such a way that it returns instantly and then runs in the background. Of course, Jenkins has no way to know how long to wait for, so problems such as those above occur.
My horrible hack around this is:
https://github.com/Simplistix/testfixtures/blob/master/jenkins.bat#L2
...which waits 20 seconds and then moves to the next step. It's far from reliable, especially when the Windows box has any load on it.
Can you make any suggestions how to solve this problem?
No, can you? :) The problem is that buildout is upgrading for a new setuptools. To do this, it needs to rewrite the buildout script, but *windows* won't let it do that because the script is open. Maybe there's a better way around this. Your workaround, if I understand it, to wait 20 seconds looks good to me. :) Another option would be to pin setuptools in your buildout.cfg to avoid the upgrade.
PS: Where does the buildout continuous integration testing live nowadays?
https://travis-ci.org/buildout/buildout Unfortunately, the buildout2 tests always fail there, even when they pass for me. I didn't set those up and I'm not sure what's going on with them. http://winbot.zope.org/builders/ Note that the buildout 1 tests always fail. (I give up.) and buildout 2 doesn't buildout on Python 3 on Windows. The buildout 2 failure looks spurious. I typically clean up the windows test failures (if I think of it) before making a release. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm
On 19/01/2013 21:16, Jim Fulton wrote:
My horrible hack around this is:
https://github.com/Simplistix/testfixtures/blob/master/jenkins.bat#L2
...which waits 20 seconds and then moves to the next step. It's far from reliable, especially when the Windows box has any load on it.
Can you make any suggestions how to solve this problem?
No, can you? :)
The problem is that buildout is upgrading for a new setuptools. To do this, it needs to rewrite the buildout script, but *windows* won't let it do that because the script is open. Maybe there's a better way around this.
Can you point me at the buildout code?
Your workaround, if I understand it, to wait 20 seconds looks good to me. :)
...except the spurious failures :-(
Another option would be to pin setuptools in your buildout.cfg to avoid the upgrade.
Would that work? How would I pin to a version of distribute?
PS: Where does the buildout continuous integration testing live nowadays?
Hmm, I don't even know how to read that I'm afraid :-S
http://winbot.zope.org/builders/
Note that the buildout 1 tests always fail. (I give up.)
I spy some 1.6.x tests, I thought the "1" branch was on 1.7 now? If those are the tests you're referring to, they appear to be failing on the git part, rather than anything further... Where can I find out how I should be looking to run the tests? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On Sat, Jan 19, 2013 at 4:51 PM, Chris Withers <chris@simplistix.co.uk> wrote:
On 19/01/2013 21:16, Jim Fulton wrote:
My horrible hack around this is:
https://github.com/Simplistix/testfixtures/blob/master/jenkins.bat#L2
...which waits 20 seconds and then moves to the next step. It's far from reliable, especially when the Windows box has any load on it.
Can you make any suggestions how to solve this problem?
No, can you? :)
The problem is that buildout is upgrading for a new setuptools. To do this, it needs to rewrite the buildout script, but *windows* won't let it do that because the script is open. Maybe there's a better way around this.
Can you point me at the buildout code?
https://github.com/buildout/buildout/blob/master/src/zc/buildout/buildout.py /me suspects a trick question. Look at _maybe_upgrade
Your workaround, if I understand it, to wait 20 seconds looks good to me. :)
...except the spurious failures :-(
Another option would be to pin setuptools in your buildout.cfg to avoid the upgrade.
Would that work? How would I pin to a version of distribute?
In buildout 2 master, you just set distribute's version in a versions section. In buildout1 you set the distribute-version option in the buildout section.
PS: Where does the buildout continuous integration testing live nowadays?
Hmm, I don't even know how to read that I'm afraid :-S
http://winbot.zope.org/builders/
Note that the buildout 1 tests always fail. (I give up.)
I spy some 1.6.x tests, I thought the "1" branch was on 1.7 now? If those are the tests you're referring to, they appear to be failing on the git part, rather than anything further...
They are now. They were failing before that switch.
Where can I find out how I should be looking to run the tests?
https://github.com/buildout/buildout/blob/master/DEVELOPERS.txt Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm
On 19/01/2013 21:16, Jim Fulton wrote:
Another option would be to pin setuptools in your buildout.cfg to avoid the upgrade.
Just a note to say that this didn't work: http://jenkins.simplistix.co.uk/job/testfixtures-buildout/PYTHON=2.7,label=w... The problem is that bootstrap.py *always* dowloads setuptools. I wish it didn't do that if the right version of setuptools was already in the egg cache... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On 19/01/2013 22:54, Chris Withers wrote:
On 19/01/2013 21:16, Jim Fulton wrote:
Another option would be to pin setuptools in your buildout.cfg to avoid the upgrade.
Just a note to say that this didn't work:
http://jenkins.simplistix.co.uk/job/testfixtures-buildout/PYTHON=2.7,label=w...
Even worse, doing so causes an infinite loop of restarts on Mac OS X, as a result of the setuptools that ships with the OS versions of python: http://jenkins.simplistix.co.uk/job/testfixtures-buildout/168/PYTHON=2.7,lab... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On Sat, Jan 19, 2013 at 6:11 PM, Chris Withers <chris@simplistix.co.uk> wrote:
On 19/01/2013 22:54, Chris Withers wrote:
On 19/01/2013 21:16, Jim Fulton wrote:
Another option would be to pin setuptools in your buildout.cfg to avoid the upgrade.
Just a note to say that this didn't work:
http://jenkins.simplistix.co.uk/job/testfixtures-buildout/PYTHON=2.7,label=w...
Even worse, doing so causes an infinite loop of restarts on Mac OS X, as a result of the setuptools that ships with the OS versions of python:
http://jenkins.simplistix.co.uk/job/testfixtures-buildout/168/PYTHON=2.7,lab...
System Python suck. (Mac OS X's Python is especially egregious because -S won't even hide extras.) But I'll look at this anyway, as infinite restarts aren't cool. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm
On 19/01/2013 23:38, Jim Fulton wrote:
Even worse, doing so causes an infinite loop of restarts on Mac OS X, as a result of the setuptools that ships with the OS versions of python:
http://jenkins.simplistix.co.uk/job/testfixtures-buildout/168/PYTHON=2.7,lab...
System Python suck. (Mac OS X's Python is especially egregious because -S won't even hide extras.)
yeah, I was wrapping them in virtualenvs, but this causes problems for tox in that virtualenv doesn't work well when a virtualenv is used as the "real python" :-(
But I'll look at this anyway, as infinite restarts aren't cool.
Thanks. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On Sat, Jan 19, 2013 at 6:11 PM, Chris Withers <chris@simplistix.co.uk> wrote: ...
Even worse, doing so causes an infinite loop of restarts on Mac OS X, as a result of the setuptools that ships with the OS versions of python:
http://jenkins.simplistix.co.uk/job/testfixtures-buildout/168/PYTHON=2.7,lab...
I just tried to reproduce this and couldn't. My failure is probably due to poor assumptions about what you're doing. (In may also be due to me still running Snow Leopard.) Looking at the link above, I can't tell what's being done. Can you demonstrate the problem without jenkins, or shining panda other scripts I can't see? Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm
On 20/01/2013 16:56, Jim Fulton wrote:
On Sat, Jan 19, 2013 at 6:11 PM, Chris Withers <chris@simplistix.co.uk> wrote: ...
Even worse, doing so causes an infinite loop of restarts on Mac OS X, as a result of the setuptools that ships with the OS versions of python:
http://jenkins.simplistix.co.uk/job/testfixtures-buildout/168/PYTHON=2.7,lab...
I just tried to reproduce this and couldn't.
See the attached .tgz, here's what I did on a mac min running 10.7.4: MiniMe:~ chris$ cd /tmp MiniMe:tmp chris$ tar xzf buildout_bug.tgz MiniMe:tmp chris$ cd buildout_bug MiniMe:buildout_bug chris$ python2.7 bootstrap.py Creating directory '/private/tmp/buildout_bug/eggs'. Getting distribution for 'setuptools==0.6c11'. Got setuptools 0.6c11. Generated script '/private/tmp/buildout_bug/bin/buildout'. MiniMe:buildout_bug chris$ bin/buildout Upgraded: setuptools version 0.6c11; restarting. Upgraded: setuptools version 0.6c11; restarting. Upgraded: setuptools version 0.6c11; restarting. Upgraded: setuptools version 0.6c11; restarting. Upgraded: setuptools version 0.6c11; restarting. ^CTraceback (most recent call last): File "bin/buildout", line 17, in <module> import zc.buildout.buildout File "/private/tmp/buildout_bug/eggs/zc.buildout-1.7.0-py2.7.egg/zc/buildout/buildout.py", line 40, in <module> import zc.buildout.download File "/private/tmp/buildout_bug/eggs/zc.buildout-1.7.0-py2.7.egg/zc/buildout/download.py", line 20, in <module> from zc.buildout.easy_install import realpath File "/private/tmp/buildout_bug/eggs/zc.buildout-1.7.0-py2.7.egg/zc/buildout/easy_install.py", line 31, in <module> import setuptools.package_index File "/Library/Python/2.7/site-packages/setuptools-0.6c12dev_r88846-py2.7.egg/setuptools/package_index.py", line 2, in <module> Traceback (most recent call last): File "bin/buildout", line 20, in <module> sys.exit(zc.buildout.buildout.main()) File "/private/tmp/buildout_bug/eggs/zc.buildout-1.7.0-py2.7.egg/zc/buildout/buildout.py", line 1866, in main File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 94, in <module> Traceback (most recent call last): File "bin/buildout", line 20, in <module> sys.exit(zc.buildout.buildout.main()) File "/private/tmp/buildout_bug/eggs/zc.buildout-1.7.0-py2.7.egg/zc/buildout/buildout.py", line 1866, in main Traceback (most recent call last): cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On Sat, Jan 19, 2013 at 11:54 PM, Chris Withers <chris@simplistix.co.uk> wrote:
On 19/01/2013 21:16, Jim Fulton wrote:
Another option would be to pin setuptools in your buildout.cfg to avoid the upgrade. Just a note to say that this didn't work: http://jenkins.simplistix.co.uk/job/testfixtures-buildout/PYTHON=2.7,label=w... The problem is that bootstrap.py *always* dowloads setuptools. I wish it didn't do that if the right version of setuptools was already in the egg cache...
You could try to pin the distribute or setuptools version used by bootstrap this way: python bootstrap.py --version 1.7.0 \ --distribute --setup-source=<path to script>/ez_setup.py \ <-- saved locally --download-base=<path to download-cache>/dist/ <-- with a saved locally version of setuptools and then have cached version of the setup and archive for distribute and setuptools, and use a buildout with 1/ pinned version of distribute or setuptools 2/ these extra buildout config info. Something along these lines: [buildout] install-from-cache = true newest = false allow-picked-versions = false download-cache = <download cache path> find-links = file:///<download cache path>/dist/ versions = versions [versions] setuptools = 0.6c11 Not sure it would help, but worth a try -- Philippe Ombredanne +1 650 799 0949 | pombredanne@nexB.com DejaCode Enterprise at http://www.dejacode.com nexB Inc. at http://www.nexb.com
On 21/01/2013 09:39, Philippe Ombredanne wrote:
On Sat, Jan 19, 2013 at 11:54 PM, Chris Withers<chris@simplistix.co.uk> wrote:
On 19/01/2013 21:16, Jim Fulton wrote:
Another option would be to pin setuptools in your buildout.cfg to avoid the upgrade. Just a note to say that this didn't work: http://jenkins.simplistix.co.uk/job/testfixtures-buildout/PYTHON=2.7,label=w... The problem is that bootstrap.py *always* dowloads setuptools. I wish it didn't do that if the right version of setuptools was already in the egg cache...
You could try to pin the distribute or setuptools version used by bootstrap this way: python bootstrap.py --version 1.7.0 \ --distribute --setup-source=<path to script>/ez_setup.py \<-- saved locally --download-base=<path to download-cache>/dist/<-- with a saved locally version of setuptools
Given the only reason I was trying to pin setuptools was to try and workaround the windows problem the subject of this message pertains to, I don't really want to do all the hacks above. Just taking the pinned version out worked for me, which is fine since it didn't help the windows situation anyway... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
participants (3)
-
Chris Withers -
Jim Fulton -
Philippe Ombredanne