Buildbot throwing a gear after default setup?
I've just downloaded and configured the Distribute buildbot and it appears that a dependency is missing somewhere. I get this (just the errors excerpted). Just wanted to capture this before I figure out what needs to be done... test_make_tarball (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... skipped 'requires zlib' test_make_zipfile (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... skipped 'Requires zlib' test_tarfile_root_owner (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... skipped 'Requires zlib' test_tarfile_vs_tar (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... skipped 'Requires zlib' ====================================================================== ERROR: test_make_archive_owner_group (distutils.tests.test_archive_util.ArchiveUtilTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/ssteiner/distutils-builbot/parts/trunk-checkout/python/ Lib/distutils/tests/test_archive_util.py", line 224, in test_make_archive_owner_group group=group) File "/home/ssteiner/distutils-builbot/parts/trunk-checkout/python/ Lib/distutils/archive_util.py", line 236, in make_archive filename = func(base_name, base_dir, **kwargs) File "/home/ssteiner/distutils-builbot/parts/trunk-checkout/python/ Lib/distutils/archive_util.py", line 163, in make_zipfile compression=zipfile.ZIP_DEFLATED) File "/home/ssteiner/distutils-builbot/parts/trunk-checkout/python/ Lib/zipfile.py", line 675, in __init__ "Compression requires the (missing) zlib module" RuntimeError: Compression requires the (missing) zlib module ---------------------------------------------------------------------- Ran 158 tests in 1.242s FAILED (errors=1, skipped=11) test test_distutils crashed -- <type 'exceptions.OSError'>: [Errno 2] No such file or directory Traceback (most recent call last): File "Lib/test/regrtest.py", line 767, in runtest_inner File "Lib/test/regrtest.py", line 723, in __exit__ File "Lib/test/regrtest.py", line 686, in get_cwd OSError: [Errno 2] No such file or directory 1 test failed: test_distutils
Looks like you're using a build of Python that wasn't compiled with zlib support (--with-zlib, IIRC). 2009/10/16 ssteinerX@gmail.com <ssteinerx@gmail.com>:
I've just downloaded and configured the Distribute buildbot and it appears that a dependency is missing somewhere. I get this (just the errors excerpted).
Just wanted to capture this before I figure out what needs to be done...
test_make_tarball (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... skipped 'requires zlib' test_make_zipfile (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... skipped 'Requires zlib' test_tarfile_root_owner (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... skipped 'Requires zlib' test_tarfile_vs_tar (distutils.tests.test_archive_util.ArchiveUtilTestCase) ... skipped 'Requires zlib'
====================================================================== ERROR: test_make_archive_owner_group (distutils.tests.test_archive_util.ArchiveUtilTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/ssteiner/distutils-builbot/parts/trunk-checkout/python/Lib/distutils/tests/test_archive_util.py", line 224, in test_make_archive_owner_group group=group) File "/home/ssteiner/distutils-builbot/parts/trunk-checkout/python/Lib/distutils/archive_util.py", line 236, in make_archive filename = func(base_name, base_dir, **kwargs) File "/home/ssteiner/distutils-builbot/parts/trunk-checkout/python/Lib/distutils/archive_util.py", line 163, in make_zipfile compression=zipfile.ZIP_DEFLATED) File "/home/ssteiner/distutils-builbot/parts/trunk-checkout/python/Lib/zipfile.py", line 675, in __init__ "Compression requires the (missing) zlib module" RuntimeError: Compression requires the (missing) zlib module
---------------------------------------------------------------------- Ran 158 tests in 1.242s
FAILED (errors=1, skipped=11) test test_distutils crashed -- <type 'exceptions.OSError'>: [Errno 2] No such file or directory Traceback (most recent call last): File "Lib/test/regrtest.py", line 767, in runtest_inner File "Lib/test/regrtest.py", line 723, in __exit__ File "Lib/test/regrtest.py", line 686, in get_cwd OSError: [Errno 2] No such file or directory 1 test failed: test_distutils
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 ssteinerX@gmail.com wrote:
On Oct 16, 2009, at 6:58 PM, Nathan Yergler wrote:
Looks like you're using a build of Python that wasn't compiled with zlib support (--with-zlib, IIRC).
Right -- that's the python that the buildbot compiled from source, that's why it's a bug ;-).
You need to install package contains the headers for zlib in your OS / distro: Python conditionally compiles in zlib support based on whether it can find the .h file. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkrY/zwACgkQ+gerLs4ltQ6NzgCfbEAgqorW/mAyCcJAinLr5RB0 ZhcAn0TF6n9SjPEfKGPYJlJ4vj0fGIhY =UIai -----END PGP SIGNATURE-----
On Sat, Oct 17, 2009 at 12:58 AM, Nathan Yergler <nathan@creativecommons.org> wrote:
Looks like you're using a build of Python that wasn't compiled with zlib support (--with-zlib, IIRC).
2009/10/16 ssteinerX@gmail.com <ssteinerx@gmail.com>:
I've just downloaded and configured the Distribute buildbot and it appears that a dependency is missing somewhere. I get this (just the errors excerpted).
Nathan is right, but this Distutils test was supposed to check if zlib is supported before its run, I have fixed it in r75450. So you should be OK now. Thanks for noticing ! Tarek
On Oct 16, 2009, at 7:04 PM, Tarek Ziadé wrote:
On Sat, Oct 17, 2009 at 12:58 AM, Nathan Yergler <nathan@creativecommons.org> wrote:
Looks like you're using a build of Python that wasn't compiled with zlib support (--with-zlib, IIRC).
2009/10/16 ssteinerX@gmail.com <ssteinerx@gmail.com>:
I've just downloaded and configured the Distribute buildbot and it appears that a dependency is missing somewhere. I get this (just the errors excerpted).
Nathan is right, but this Distutils test was supposed to check if zlib is supported before its run,
I have fixed it in r75450. So you should be OK now.
Thanks for noticing !
Cool, no problem. I also reported it in the buildbot project on BitBucket. Odd thing is: ssteiner@ubuntu:~/distutils-builbot$ cat bin/test #!/usr/bin/python # -*- coding: utf-8 -*- ... ssteiner@ubuntu:~/distutils-builbot$ /usr/bin/python Python 2.6.2 (release26-maint, Apr 19 2009, 01:58:18) [GCC 4.3.3] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import zlib
works fine, so I'm not sure why it thinks it's not able to get zlib. S
On Sat, Oct 17, 2009 at 1:13 AM, ssteinerX@gmail.com <ssteinerx@gmail.com> wrote:
On Oct 16, 2009, at 7:04 PM, Tarek Ziadé wrote:
On Sat, Oct 17, 2009 at 12:58 AM, Nathan Yergler <nathan@creativecommons.org> wrote:
Looks like you're using a build of Python that wasn't compiled with zlib support (--with-zlib, IIRC).
2009/10/16 ssteinerX@gmail.com <ssteinerx@gmail.com>:
I've just downloaded and configured the Distribute buildbot and it appears that a dependency is missing somewhere. I get this (just the errors excerpted).
Nathan is right, but this Distutils test was supposed to check if zlib is supported before its run,
I have fixed it in r75450. So you should be OK now.
Thanks for noticing !
Cool, no problem. I also reported it in the buildbot project on BitBucket.
Odd thing is:
ssteiner@ubuntu:~/distutils-builbot$ cat bin/test #!/usr/bin/python # -*- coding: utf-8 -*- ...
ssteiner@ubuntu:~/distutils-builbot$ /usr/bin/python Python 2.6.2 (release26-maint, Apr 19 2009, 01:58:18) [GCC 4.3.3] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import zlib
works fine, so I'm not sure why it thinks it's not able to get zlib.
Python gets build from scratch everytime here, and "test" just runs a subprocess to call the "other" python located in the buildbot. IOW the way python is built in the bbot, is now using your system's zlib support... I need to check why... Regards Tarek
S
-- Tarek Ziadé | http://ziade.org | オープンソースはすごい! | 开源传万世，因有你参与
On Sat, Oct 17, 2009 at 1:21 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
IOW the way python is built in the bbot, is now using your system's zlib support...
Arggg... I am the typo master... s/now/not. So as Tres said maybe Ubuntu ships with a Python with zlib support, but doesn't provide zlib headers.
On Oct 16, 2009, at 7:26 PM, Tarek Ziadé wrote:
On Sat, Oct 17, 2009 at 1:21 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
IOW the way python is built in the bbot, is now using your system's zlib support...
Arggg... I am the typo master...
s/now/not.
So as Tres said maybe Ubuntu ships with a Python with zlib support, but doesn't provide zlib headers.
Ok, so the python you build from the trunk checkout doesn't have zlib support. I couldn't find an apt-get'able one, it looks like you have to build from source, but the lack of zlib support should be picked up before the unit tests fail. S
On Sat, Oct 17, 2009 at 1:39 AM, ssteinerX@gmail.com <ssteinerx@gmail.com> wrote:
On Oct 16, 2009, at 7:26 PM, Tarek Ziadé wrote:
On Sat, Oct 17, 2009 at 1:21 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
IOW the way python is built in the bbot, is now using your system's zlib support...
Arggg... I am the typo master...
s/now/not.
So as Tres said maybe Ubuntu ships with a Python with zlib support, but doesn't provide zlib headers.
Ok, so the python you build from the trunk checkout doesn't have zlib support.
Yup.
I couldn't find an apt-get'able one, it looks like you have to build from source, but the lack of zlib support should be picked up before the unit tests fail.
Yes that's what I've fixed, so it should not fail anymore. I'll digg in zlib@ubuntu support in any case.
On Oct 16, 2009, at 7:57 PM, Tarek Ziadé wrote:
On Sat, Oct 17, 2009 at 1:39 AM, ssteinerX@gmail.com <ssteinerx@gmail.com> wrote:
On Oct 16, 2009, at 7:26 PM, Tarek Ziadé wrote:
On Sat, Oct 17, 2009 at 1:21 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
IOW the way python is built in the bbot, is now using your system's zlib support...
Arggg... I am the typo master...
s/now/not.
So as Tres said maybe Ubuntu ships with a Python with zlib support, but doesn't provide zlib headers.
Ok, so the python you build from the trunk checkout doesn't have zlib support.
Yup.
I couldn't find an apt-get'able one, it looks like you have to build from source, but the lack of zlib support should be picked up before the unit tests fail.
Yes that's what I've fixed, so it should not fail anymore.
Where can I get that? I pulled into distutils-buildbot and nothing came down.
I'll digg in zlib@ubuntu support in any case.
ssteiner@ubuntu:~/distutils-builbot$ aptitude search zlib p zlib-bin - compression library - sample programs p zlib-gst - Zlib bindings for GNU Smalltalk i zlib1g - compression library - runtime p zlib1g-dbg - compression library - development p zlib1g-dev - compression library - development p zlibc - An on- fly auto-uncompressing C library # sudo aptitude install zlib-bin zlib-dev zlib-dbg zlibc Not sure if they're all needed... l'll rebuild as soon as I know where to get the changes. S
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 ssteinerX@gmail.com wrote:
On Oct 16, 2009, at 7:26 PM, Tarek Ziadé wrote:
So as Tres said maybe Ubuntu ships with a Python with zlib support, but doesn't provide zlib headers.
Ok, so the python you build from the trunk checkout doesn't have zlib support.
I couldn't find an apt-get'able one, it looks like you have to build from source, but the lack of zlib support should be picked up before the unit tests fail.
$ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=9.04 DISTRIB_CODENAME=jaunty DISTRIB_DESCRIPTION="Ubuntu 9.04" $ apt-cache search zlib | grep ^zlib.*dev zlib1g-dbg - compression library - development zlib1g-dev - compression library - development Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkrZLbQACgkQ+gerLs4ltQ6RTQCeIDYNZNJ/xp5owC4xva1wCtPP jqEAnAim9iziE6rXbHtoxvsy94ZUNis7 =MxaL -----END PGP SIGNATURE-----
Yes, I found those and installed -- not sure if the message went to the list, too. I'm waiting on how to get updated code from Tarek to re-run the whole thing. Thanks, S On Oct 16, 2009, at 10:36 PM, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
ssteinerX@gmail.com wrote:
On Oct 16, 2009, at 7:26 PM, Tarek Ziadé wrote:
So as Tres said maybe Ubuntu ships with a Python with zlib support, but doesn't provide zlib headers.
Ok, so the python you build from the trunk checkout doesn't have zlib support.
I couldn't find an apt-get'able one, it looks like you have to build from source, but the lack of zlib support should be picked up before the unit tests fail.
$ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=9.04 DISTRIB_CODENAME=jaunty DISTRIB_DESCRIPTION="Ubuntu 9.04" $ apt-cache search zlib | grep ^zlib.*dev zlib1g-dbg - compression library - development zlib1g-dev - compression library - development
Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iEYEARECAAYFAkrZLbQACgkQ+gerLs4ltQ6RTQCeIDYNZNJ/xp5owC4xva1wCtPP jqEAnAim9iziE6rXbHtoxvsy94ZUNis7 =MxaL -----END PGP SIGNATURE-----
On Sat, Oct 17, 2009 at 4:58 AM, ssteinerX@gmail.com <ssteinerx@gmail.com> wrote:
Yes, I found those and installed -- not sure if the message went to the list, too.
I'm waiting on how to get updated code from Tarek to re-run the whole thing.
Hey, My update was done on python trunk. It's a buildbot so once your buildout is created, (python bootstrap.py && bin/buildout) you have new binaries in bin/: - one to launch the buildbot server (bin/master) - one to launch the buildbot slave on your machine (bin/ziade.org) once they are launched (daemons), you can go to http://localhost:9080/. The slave will automatically run the tests by: - doing "svn up" on the python trunk (that's where you will get the change I've done on Python trunk) - building python (configure + make) - run the python tests via bin/test Let me know how it goes.
On Oct 17, 2009, at 4:54 AM, Tarek Ziadé wrote:
On Sat, Oct 17, 2009 at 10:34 AM, Tarek Ziadé <ziade.tarek@gmail.com> wrote:
It's a buildbot so once your buildout is created, (python bootstrap.py && bin/buildout)
precisely: python bootstrap.py && bin/buildout -c buildbot.cfg
ssteiner@ubuntu:~/distutils-builbot$ python bootstrap.py && bin/ buildout -c buildbot.cfg Downloading http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c9-py2.6.egg Getting distribution for 'Twisted'. twisted/runner/portmap.c:10:20: error: Python.h: No such file or directory twisted/runner/portmap.c:14: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token twisted/runner/portmap.c:31: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token twisted/runner/portmap.c:45: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘PortmapMethods’ twisted/runner/portmap.c: In function ‘initportmap’: twisted/runner/portmap.c:55: warning: implicit declaration of function ‘Py_InitModule’ twisted/runner/portmap.c:55: error: ‘PortmapMethods’ undeclared (first use in this function) twisted/runner/portmap.c:55: error: (Each undeclared identifier is reported only once twisted/runner/portmap.c:55: error: for each function it appears in.) error: Setup script exited with error: command 'gcc' failed with exit status 1 An error occured when trying to install Twisted 8.2.0.Look above this message for any errors thatwere output by easy_install. While: Installing. Getting section master. Initializing section master. Installing recipe collective.buildbot. Getting distribution for 'Twisted'. Error: Couldn't install: Twisted 8.2.0
On Sat, Oct 17, 2009 at 4:49 PM, ssteinerX@gmail.com <ssteinerx@gmail.com> wrote:
precisely: python bootstrap.py && bin/buildout -c buildbot.cfg
ssteiner@ubuntu:~/distutils-builbot$ python bootstrap.py && bin/buildout -c buildbot.cfg Downloading http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c9-py2.6.egg Getting distribution for 'Twisted'. twisted/runner/portmap.c:10:20: error: Python.h: No such file or directory
Buildbot needs Twisted, so it needs the 'python-dev' to compile on ubuntu. $ apt-get install python-dev $ apt-get install build-essentials (should be enough) Tarek.
On Oct 17, 2009, at 12:24 PM, Tarek Ziadé wrote:
On Sat, Oct 17, 2009 at 4:49 PM, ssteinerX@gmail.com <ssteinerx@gmail.com> wrote:
precisely: python bootstrap.py && bin/buildout -c buildbot.cfg
ssteiner@ubuntu:~/distutils-builbot$ python bootstrap.py && bin/ buildout -c buildbot.cfg Downloading http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c9-py2.6.egg Getting distribution for 'Twisted'. twisted/runner/portmap.c:10:20: error: Python.h: No such file or directory
Buildbot needs Twisted, so it needs the 'python-dev' to compile on ubuntu.
$ apt-get install python-dev $ apt-get install build-essentials
Is there an easy way of getting bootstrap.py to run these (and get Zlib) on Ubuntu since they're not installed on a default image? S
On Oct 17, 2009, at 12:24 PM, Tarek Ziadé wrote:
On Sat, Oct 17, 2009 at 4:49 PM, ssteinerX@gmail.com <ssteinerx@gmail.com> wrote:
precisely: python bootstrap.py && bin/buildout -c buildbot.cfg
ssteiner@ubuntu:~/distutils-builbot$ python bootstrap.py && bin/ buildout -c buildbot.cfg Downloading http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c9-py2.6.egg Getting distribution for 'Twisted'. twisted/runner/portmap.c:10:20: error: Python.h: No such file or directory
Buildbot needs Twisted, so it needs the 'python-dev' to compile on ubuntu.
$ apt-get install python-dev $ apt-get install build-essentials (should be enough)
That got me further (through installing Twisted) but now: # ssteiner@ubuntu:~/distutils-builbot$ python bootstrap.py && bin/ buildout -c buildbot.cfg Downloading http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c9-py2.6.egg Getting distribution for 'Twisted'. [Twisted builds...] Got Twisted 8.2.0. Getting distribution for 'virtualenv'. Got virtualenv 1.3.4. Getting distribution for 'zc.recipe.egg'. Got zc.recipe.egg 1.2.2. Getting distribution for 'PasteDeploy'. warning: no files found matching 'docs/*.html' warning: no previously-included files found matching 'docs/rebuild' Got PasteDeploy 1.3.3. Getting distribution for 'Paste>=1.3'. Got Paste 1.7.2. While: Installing. Getting section master. Initializing section master. Loading zc.buildout recipe entry collective.buildbot:master. An internal error occured due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): File "/home/ssteiner/distutils-builbot/eggs/zc.buildout-1.4.1- py2.6.egg/zc/buildout/buildout.py", line 1659, in main getattr(buildout, command)(args) File "/home/ssteiner/distutils-builbot/eggs/zc.buildout-1.4.1- py2.6.egg/zc/buildout/buildout.py", line 416, in install [self[part]['recipe'] for part in install_parts] File "/home/ssteiner/distutils-builbot/eggs/zc.buildout-1.4.1- py2.6.egg/zc/buildout/buildout.py", line 963, in __getitem__ options._initialize() File "/home/ssteiner/distutils-builbot/eggs/zc.buildout-1.4.1- py2.6.egg/zc/buildout/buildout.py", line 1047, in _initialize recipe_class = _install_and_load(reqs, 'zc.buildout', entry, buildout) File "/home/ssteiner/distutils-builbot/eggs/zc.buildout-1.4.1- py2.6.egg/zc/buildout/buildout.py", line 1008, in _install_and_load req.project_name, group, entry) File "/home/ssteiner/distutils-builbot/eggs/setuptools-0.6c9- py2.6.egg/pkg_resources.py", line 277, in load_entry_point return get_distribution(dist).load_entry_point(group, name) File "/home/ssteiner/distutils-builbot/eggs/setuptools-0.6c9- py2.6.egg/pkg_resources.py", line 2180, in load_entry_point return ep.load() File "/home/ssteiner/distutils-builbot/eggs/setuptools-0.6c9- py2.6.egg/pkg_resources.py", line 1913, in load entry = __import__(self.module_name, globals(),globals(), ['__name__']) File "/home/ssteiner/distutils-builbot/eggs/ collective.buildbot-0.3.5-py2.6.egg/collective/buildbot/__init__.py", line 48, in <module> from buildbot.steps.shell import WarningCountingShellCommand File "/usr/local/lib/python2.6/dist-packages/buildbot/steps/ shell.py", line 5, in <module> from buildbot.process.buildstep import LoggingBuildStep, RemoteShellCommand File "/usr/local/lib/python2.6/dist-packages/buildbot/process/ buildstep.py", line 9, in <module> from twisted.web.util import formatFailure ImportError: No module named web.util ssteiner@ubuntu:~/distutils-builbot$ I guess it's a good thing we're doing this! ;-O S
On Oct 17, 2009, at 12:24 PM, Tarek Ziadé wrote: Just to be clear...I'm not just giving up when the build falls down, I just do want to make sure to report everything I do so that we can get the buildbot to configure whatever's not there automatically instead of having to do anything manually. We want to be able to fire these up, on a remote, freshly minted Cloud Server, and have it pull in everything it needs to run with no operator intervention, run tests, and self-destruct all within less than an hour (i.e. 1.5 cents worth of compute power). Right? S P.S. To get Twisted to build, I had to: # sudo apt-get install zlib1g-dev zlib1g-dev s
On Sat, Oct 17, 2009 at 8:53 PM, ssteinerX@gmail.com <ssteinerx@gmail.com> wrote:
On Oct 17, 2009, at 12:24 PM, Tarek Ziadé wrote:
Just to be clear...I'm not just giving up when the build falls down, I just do want to make sure to report everything I do so that we can get the buildbot to configure whatever's not there automatically instead of having to do anything manually.
We want to be able to fire these up, on a remote, freshly minted Cloud Server, and have it pull in everything it needs to run with no operator intervention, run tests, and self-destruct all within less than an hour (i.e. 1.5 cents worth of compute power).
Right?
Right, but my buildbot made some assumptions on what's required on the platform (you noticed ;) ) Now I don't understand your last error from twisted.web.util import formatFailure
ImportError: No module named web.util
Since Twisted seemed to be added correctly I think we should meet online at #distutils to tackle this. Tarek
On 04:24 pm, ziade.tarek@gmail.com wrote:
On Sat, Oct 17, 2009 at 4:49 PM, ssteinerX@gmail.com <ssteinerx@gmail.com> wrote:
precisely: python bootstrap.py && bin/buildout -c buildbot.cfg
ssteiner@ubuntu:~/distutils-builbot$ python bootstrap.py && bin/buildout -c buildbot.cfg Downloading http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c9-py2.6.egg Getting distribution for 'Twisted'. twisted/runner/portmap.c:10:20: error: Python.h: No such file or directory
Buildbot needs Twisted, so it needs the 'python-dev' to compile on ubuntu.
$ apt-get install python-dev $ apt-get install build-essentials (should be enough)
Almost all of Twisted's extension modules are optional (and portmap is hardly useful, I doubt anyone would miss it if it were gone). This failure reminds me that it would be nice if distutils had a way to express this. Jean-Paul
On Oct 17, 2009, at 3:47 PM, exarkun@twistedmatrix.com wrote:
On 04:24 pm, ziade.tarek@gmail.com wrote:
On Sat, Oct 17, 2009 at 4:49 PM, ssteinerX@gmail.com <ssteinerx@gmail.com> wrote:
precisely: python bootstrap.py && bin/buildout -c buildbot.cfg
ssteiner@ubuntu:~/distutils-builbot$ python bootstrap.py && bin/ buildout -c buildbot.cfg Downloading http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c9-py2.6.egg Getting distribution for 'Twisted'. twisted/runner/portmap.c:10:20: error: Python.h: No such file or directory
Buildbot needs Twisted, so it needs the 'python-dev' to compile on ubuntu.
$ apt-get install python-dev $ apt-get install build-essentials (should be enough)
Almost all of Twisted's extension modules are optional (and portmap is hardly useful, I doubt anyone would miss it if it were gone). This failure reminds me that it would be nice if distutils had a way to express this.
So, is this something I have to install in addition to the way Twisted's being installed now? I'm working on setting up the buildbot automatically on a cloud server and all these errors are what happen if you start from a plain-vanilla Ubuntu 9.0.4 and try to just bootstrap the distutils buildbot. S
On 17 Oct, 11:26 pm, ssteinerx@gmail.com wrote:
On Oct 17, 2009, at 3:47 PM, exarkun@twistedmatrix.com wrote:
On 04:24 pm, ziade.tarek@gmail.com wrote:
On Sat, Oct 17, 2009 at 4:49 PM, ssteinerX@gmail.com <ssteinerx@gmail.com> wrote:
precisely: python bootstrap.py && bin/buildout -c buildbot.cfg
ssteiner@ubuntu:~/distutils-builbot$ python bootstrap.py && bin/ buildout -c buildbot.cfg Downloading http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c9-py2.6.egg Getting distribution for 'Twisted'. twisted/runner/portmap.c:10:20: error: Python.h: No such file or directory
Buildbot needs Twisted, so it needs the 'python-dev' to compile on ubuntu.
$ apt-get install python-dev $ apt-get install build-essentials (should be enough)
Almost all of Twisted's extension modules are optional (and portmap is hardly useful, I doubt anyone would miss it if it were gone). This failure reminds me that it would be nice if distutils had a way to express this.
So, is this something I have to install in addition to the way Twisted's being installed now?
portmap? No, it's an extension module that's part of Twisted. Once you have python-dev installed it should be okay. Jean-Paul
On Oct 18, 2009, at 11:26 AM, exarkun@twistedmatrix.com wrote:
So, is this something I have to install in addition to the way Twisted's being installed now?
portmap? No, it's an extension module that's part of Twisted. Once you have python-dev installed it should be okay.
Yah, I'm thinking that this is supposed to be building against the Python 2.7 that's built in the course of building the buildbot so I think the bug has less to do with the host environment and more to do with the buildbot not using its own self-contained stuff to build what it needs but I haven't looked too closely yet. S
On Oct 17, 2009, at 4:34 AM, Tarek Ziadé wrote:
On Sat, Oct 17, 2009 at 4:58 AM, ssteinerX@gmail.com <ssteinerx@gmail.com> wrote:
Yes, I found those and installed -- not sure if the message went to the list, too.
I'm waiting on how to get updated code from Tarek to re-run the whole thing.
Hey,
My update was done on python trunk.
Ah, wasn't sure where to get it; I wasn't sure whether you'd changed the buildbot code on bitbucket or something else.
It's a buildbot so once your buildout is created, (python bootstrap.py && bin/buildout) you have new binaries in bin/:
- one to launch the buildbot server (bin/master) - one to launch the buildbot slave on your machine (bin/ziade.org)
once they are launched (daemons), you can go to http://localhost: 9080/.
The slave will automatically run the tests by:
- doing "svn up" on the python trunk (that's where you will get the change I've done on Python trunk) - building python (configure + make) - run the python tests via bin/test
bin/test was what complained about missing zlib. I ran it by hand to see if it was working.
Let me know how it goes.
I thought I had apt-get installed all the zlib components so I wasn't expecting problems but I guess not. It now throws a different gear in a different way, so I guess I got your changes :-) Creating tar archive Traceback (most recent call last): File "setup.py", line 15, in <module> packages=['extensions', 'extensions.command'] File "/home/ssteiner/distutils-builbot/parts/trunk-checkout/python/ Lib/distutils/core.py", line 149, in setup dist.run_commands() File "/home/ssteiner/distutils-builbot/parts/trunk-checkout/python/ Lib/distutils/dist.py", line 926, in run_commands self.run_command(cmd) File "/home/ssteiner/distutils-builbot/parts/trunk-checkout/python/ Lib/distutils/dist.py", line 945, in run_command cmd_obj.run() File "/home/ssteiner/distutils-builbot/parts/trunk-checkout/python/ Lib/distutils/command/sdist.py", line 166, in run self.make_distribution() File "/home/ssteiner/distutils-builbot/parts/trunk-checkout/python/ Lib/distutils/command/sdist.py", line 465, in make_distribution owner=self.owner, group=self.group) File "/home/ssteiner/distutils-builbot/parts/trunk-checkout/python/ Lib/distutils/cmd.py", line 392, in make_archive owner=owner, group=group) File "/home/ssteiner/distutils-builbot/parts/trunk-checkout/python/ Lib/distutils/archive_util.py", line 236, in make_archive filename = func(base_name, base_dir, **kwargs) File "/home/ssteiner/distutils-builbot/parts/trunk-checkout/python/ Lib/distutils/archive_util.py", line 101, in make_tarball tar = tarfile.open(archive_name, 'w|%s' % tar_compression [compress]) File "/home/ssteiner/distutils-builbot/parts/trunk-checkout/python/ Lib/tarfile.py", line 1668, in open _Stream(name, filemode, comptype, fileobj, bufsize), File "/home/ssteiner/distutils-builbot/parts/trunk-checkout/python/ Lib/tarfile.py", line 417, in __init__ raise CompressionError("zlib module is not available") tarfile.CompressionError: zlib module is not available Exception AttributeError: "_Stream instance has no attribute 'cmp'" in <bound method _Stream.__del__ of <tarfile._Stream instance at 0x7fadedc8ec68>> ignored Could not create distro with /home/ssteiner/distutils-builbot/parts/ trunk-checkout/python/python2.7
On Sat, Oct 17, 2009 at 4:47 PM, ssteinerX@gmail.com <ssteinerx@gmail.com> wrote:
Let me know how it goes.
I thought I had apt-get installed all the zlib components so I wasn't expecting problems but I guess not. It now throws a different gear in a different way, so I guess I got your changes :-)
Creating tar archive Traceback (most recent call last): File "setup.py", line 15, in <module> packages=['extensions', 'extensions.command'] File
Yes, unfortunaltely this buildbot step requires zlib to build a tar.gz... What we can do is change this test here (on distutils-buildbot side) so all tests are made with another archive format. Tarek
On Oct 16, 2009, at 7:21 PM, Tarek Ziadé wrote:
On Sat, Oct 17, 2009 at 1:13 AM, ssteinerX@gmail.com <ssteinerx@gmail.com> wrote:
On Oct 16, 2009, at 7:04 PM, Tarek Ziadé wrote:
On Sat, Oct 17, 2009 at 12:58 AM, Nathan Yergler <nathan@creativecommons.org> wrote:
Looks like you're using a build of Python that wasn't compiled with zlib support (--with-zlib, IIRC).
2009/10/16 ssteinerX@gmail.com <ssteinerx@gmail.com>:
I've just downloaded and configured the Distribute buildbot and it appears that a dependency is missing somewhere. I get this (just the errors excerpted).
Nathan is right, but this Distutils test was supposed to check if zlib is supported before its run,
I have fixed it in r75450. So you should be OK now.
Thanks for noticing !
Cool, no problem. I also reported it in the buildbot project on BitBucket.
Odd thing is:
ssteiner@ubuntu:~/distutils-builbot$ cat bin/test #!/usr/bin/python # -*- coding: utf-8 -*- ...
ssteiner@ubuntu:~/distutils-builbot$ /usr/bin/python Python 2.6.2 (release26-maint, Apr 19 2009, 01:58:18) [GCC 4.3.3] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import zlib
works fine, so I'm not sure why it thinks it's not able to get zlib.
Python gets build from scratch everytime here, and "test" just runs a subprocess to call the "other" python located in the buildbot.
IOW the way python is built in the bbot, is now using your system's zlib support...
No, the Python I called was the system python in /usr/bin, I see that test is grabbing the trunk checkout python in python = os.path.join (root, 'parts', 'trunk-checkout', 'python', 'python') which is NOT built with zlib. The error is in the build of the trunk checkout -- zlib is in my system python just fine; it's just not in the python built from the trunk checkout: ssteiner@ubuntu:~/distutils-builbot$ parts/trunk-checkout/python/python Python 2.7a0 (trunk:75449, Oct 16 2009, 16:08:27) [GCC 4.3.3] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import zlib Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named zlib
S
participants (5)
-
exarkun＠twistedmatrix.com -
Nathan Yergler -
ssteinerX＠gmail.com -
Tarek Ziadé -
Tres Seaver