zc.buildout: Build python in buildout
Hey, building python with buildout is not very hard (see below). When installing sections buildout apparently uses the built python to get dependencies and compile the C extensions if any. Only the dependencies of the recipes are fetched via the bootstrap python. But with C extensions this is the problem. For instance zc.zodbrecipes depend on ZODB3 which is then compiled with the wrong python. This would not be much of a problem but I then get those errors: ImportError: /tmp/test/eggs/ZODB3-3.8.0b4-py2.4-linux-x86_64.egg/persistent/cPersistence.so: undefined symbol: PyUnicodeUCS4_AsEncodedString Any clue what to do? I suppose the only way is to make the two pythons eat the same .so files? This does the trick of building python: [buildout] python = python [python] recipe = zc.recipe.cmmi url = http://python.org/ftp/python/2.4.4/Python-2.4.4.tar.bz2 executable = ${buildout:directory}/parts/python/bin/python2.4 -- Christian Zagrodnick gocept gmbh & co. kg · forsterstrasse 29 · 06112 halle/saale www.gocept.com · fon. +49 345 12298894 · fax. +49 345 12298891
On Nov 28, 2007, at 12:41 PM, Christian Zagrodnick wrote:
Hey,
building python with buildout is not very hard (see below).
When installing sections buildout apparently uses the built python to get dependencies and compile the C extensions if any.
Only the dependencies of the recipes are fetched via the bootstrap python. But with C extensions this is the problem. For instance zc.zodbrecipes depend on ZODB3 which is then compiled with the wrong python.
This would not be much of a problem but I then get those errors:
ImportError: /tmp/test/eggs/ZODB3-3.8.0b4-py2.4-linux-x86_64.egg/persistent/ cPersistence.so: undefined symbol: PyUnicodeUCS4_AsEncodedString
Ouch. I guess the Python used to run the buildout and the Python built by the buildout are the same Python version.
Any clue what to do?
I have clues, although I'm not sure they'll be useful. If the Python used to run the buildout and the Python built by the buildout were different versions, then you'd be OK. I suspect that a better solution would be to find a way to bootstrap the buildout in a way that included building Python as part of the bootstrapping process.
I suppose the only way is to make the two pythons eat the same .so files?
I don't know how you'd do that. Jim -- Jim Fulton Zope Corporation
On 2007-11-28 21:48:13 +0100, Jim Fulton <jim@zope.com> said:
On Nov 28, 2007, at 12:41 PM, Christian Zagrodnick wrote:
Hey,
building python with buildout is not very hard (see below).
When installing sections buildout apparently uses the built python to get dependencies and compile the C extensions if any.
Only the dependencies of the recipes are fetched via the bootstrap python. But with C extensions this is the problem. For instance zc.zodbrecipes depend on ZODB3 which is then compiled with the wrong python.
This would not be much of a problem but I then get those errors:
ImportError: /tmp/test/eggs/ZODB3-3.8.0b4-py2.4-linux-x86_64.egg/persistent/ cPersistence.so: undefined symbol: PyUnicodeUCS4_AsEncodedString
Ouch.
I guess the Python used to run the buildout and the Python built by the buildout are the same Python version.
Yes.
Any clue what to do?
I have clues, although I'm not sure they'll be useful. If the Python used to run the buildout and the Python built by the buildout were different versions, then you'd be OK.
Yes. That was my first guess, too. The problem is that there is only a sort of broken python2.4 available there. So I had to come up with something else. :/
I suspect that a better solution would be to find a way to bootstrap the buildout in a way that included building Python as part of the bootstrapping process.
That would certainly the best approach. Only that I have no idea how to do that. :(
I suppose the only way is to make the two pythons eat the same .so files?
I don't know how you'd do that.
Ubuntu's Python is compiled with --enable-uncide=ucs4 which leads to the incompatibiliby. When compiling the Python in buildout with this option everything works fine (so far anyway). So, no, there is no general approach to fix the incompatibility, yet. Only on specific for this one Server. Anyway, thanks for the ideas. Regards, -- Christian Zagrodnick gocept gmbh & co. kg · forsterstrasse 29 · 06112 halle/saale www.gocept.com · fon. +49 345 12298894 · fax. +49 345 12298891
On Nov 29, 2007, at 2:04 AM, Christian Zagrodnick wrote:
On 2007-11-28 21:48:13 +0100, Jim Fulton <jim@zope.com> said:
I suspect that a better solution would be to find a way to bootstrap the buildout in a way that included building Python as part of the bootstrapping process.
That would certainly the best approach. Only that I have no idea how to do that. :(
Write a variation on the standard bootstrap script that first builds Python. It could do so directly, or run run buildout to do it. Then, after building Python, redo the bootstrap using this new Python. You could do this by invoking the buildout with the new Python: path_to_new_python bin/buildout bootstrap Jim -- Jim Fulton Zope Corporation
Christian Zagrodnick wrote:
On 2007-11-28 21:48:13 +0100, Jim Fulton <jim@zope.com> said:
I suspect that a better solution would be to find a way to bootstrap the buildout in a way that included building Python as part of the bootstrapping process.
That would certainly the best approach. Only that I have no idea how to do that. :(
I'm not so sure. I'd prefer bootstrapping not to look too deeply into the buildout configuration, just to keep it simple. Couldn't buildout be taught to restart itself using a custom Python if it finds that one is used, possibly after it has been rebuilt? -- Thomas
On Nov 30, 2007, at 12:42 AM, Thomas Lotze wrote:
Christian Zagrodnick wrote:
On 2007-11-28 21:48:13 +0100, Jim Fulton <jim@zope.com> said:
I suspect that a better solution would be to find a way to bootstrap the buildout in a way that included building Python as part of the bootstrapping process.
That would certainly the best approach. Only that I have no idea how to do that. :(
I'm not so sure. I'd prefer bootstrapping not to look too deeply into the buildout configuration, just to keep it simple.
Couldn't buildout be taught to restart itself using a custom Python if it finds that one is used, possibly after it has been rebuilt?
Possibly. So if a buildout python option points to a part (a section with a recipe), then it might re-write its script and restart itself if it isn't already running the Python defined by that part. Jim -- Jim Fulton Zope Corporation
Jim Fulton wrote:
Possibly. So if a buildout python option points to a part (a section with a recipe), then it might re-write its script and restart itself if it isn't already running the Python defined by that part.
No, I don't mean for buildout to rewrite its own script. I mean that the unchanged buildout script should call path_to_new_python bin/buildout $options Running the buildout script should always invoke the Python used for bootstrapping. Otherwise I fear it's possible to screw up the whole buildout by misconfiguring the Python part. -- Thomas
I suspect that a better solution would be to find a way to bootstrap the buildout in a way that included building Python as part of the bootstrapping process.
That would certainly the best approach. Only that I have no idea how to do that. :(
Examples: http://source.schooltool.org/viewcvs/trunk/st-buildout/?rev=7260#dirlist - uses a shell script that builds python using buildout, and then builds the product using the new python http://source.schooltool.org/viewcvs/trunk/st-buildout/ - uses a patched bootstrap.py that builds virtual python and uses the virtual python to bootstrap buildout hope that helps Ignas
On 2007-11-30 16:41:05 +0100, "Ignas Mikalajunas" <ignas.mikalajunas@gmail.com> said:
I suspect that a better solution would be to find a way to bootstrap the buildout in a way that included building Python as part of the bootstrapping process.
That would certainly the best approach. Only that I have no idea how to do that. :(
Examples:
http://source.schooltool.org/viewcvs/trunk/st-buildout/?rev=7260#dirlist - uses a shell script that builds python using buildout, and then builds the product using the new python
http://source.schooltool.org/viewcvs/trunk/st-buildout/ - uses a patched bootstrap.py that builds virtual python and uses the virtual python to bootstrap buildout
hope that helps
Yes, thanks for the pointers. -- Christian Zagrodnick gocept gmbh & co. kg · forsterstrasse 29 · 06112 halle/saale www.gocept.com · fon. +49 345 12298894 · fax. +49 345 12298891
participants (4)
-
Christian Zagrodnick -
Ignas Mikalajunas -
Jim Fulton -
Thomas Lotze