[Distutils] using python generated by buildout to run buildout

[Łukasz Nowak]

Hello,

I am playing with zc.buildout for some time. I experienced issues with
system provided pythons (even virutalenved), so I switched to build my
own python. Of course as I am lazy, I am using buildout to configure
and compile various python versions on my machine.

Quite often I need to provide "self contained" buildout profile, which
would be able to run using python compiled by itself.

So my example profile looks like this:

[buildout]
parts =
  rebootstrap2.4
  realrun

find-links = http://www.nexedi.org/static/packages/source/

[python2.4]
prefix = ${buildout:parts-directory}/${:_buildout_section_name_}
executable = ${:prefix}/bin/python2.4

# this is just wrapper for hexagonit.recipe.cmmi which makes it not
returning python part
recipe = erp5.recipe.cmmisafe
url =
  http://python.org/ftp/python/2.4.6/Python-2.4.6.tgz
configure-options =
  --enable-unicode=ucs4
  --with-threads
  --with-readline
  --with-dbm
  --with-zlib
  --with-ssl
  --with-bz2

[rebootstrap2.4]
recipe = zc.recipe.egg
eggs = zc.buildout
python = python2.4

[realrun]
recipe = plone.recipe.command
command =
  echo Running with python ${buildout:executable}
update-command = ${:command}

So first step for me is to bootstrap builodut:

curl -s http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py
| python -S -

So now my bin/buildout is referring to "system python":
head -n1 bin/buildout
#!/home/luke/bin/python -S

Then I need to rebootstrap it:

bin/buildout install python2.4 rebootstrap2.4

python2.4* parts are installing and compiling python, rebootstrap2.4
part is reinstalling bin/buildout using this python.

Cool, my bin/buildout is using python it compiled:

head -n1 bin/buildout
#!/home/luke/pybuildout/parts/python2.4/bin/python2.4

So I can run again, this time usually. python will be compiled, as
buildout signature changed, but this is not an issue (ccache + distcc
makes it really fast). Buildout is using python he provided,
chicken-and-egg issue solved by hand:

bin/buildout
...

realrun: Running '
echo Running with python /home/luke/pybuildout/parts/python2.4/bin/python2.4'
Running with python /home/luke/pybuildout/parts/python2.4/bin/python2.4
Unused options for realrun: 'update-command'.

Re running builodut does not affect python anymore:

bin/buildout
Updating python2.4-dbm-patch.
Updating python2.4.
Updating rebootstrap2.4.
Updating realrun.
realrun: Running
echo Running with python /home/luke/pybuildout/parts/python2.4/bin/python2.4
Running with python /home/luke/pybuildout/parts/python2.4/bin/python2.4

And so far I am happy, I am using correct python version.

What I'd like to develop is to make those steps automatically, so that
after running:

curl -s http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py
| python -S -
bin/buildout

Such steps would happen automagically:

 1 normal bootstrap with system python
 2 compile python as needed
 3 install new bin/buildout
 4 re-run bin/buildout
 5 compile python as signature change
 6 install new bin/buildout (which is the same)
 7 re-run bin/buildout
 8 do not compile python, as signature had not changed, continue...

Theoretically steps 5,6 and 7 could be avoided, but lets say I am
purist -- I do not trust that python compiled by buildout which is
running python I do not trust is bad :)

Easy question: Is there ability for buildout to do smart rebootstrap
in one run, with selecting part which contains python?

If no I am ready to develop such thing. So I read a bit about
extensions - I can hook before and after buildout is run. Extensions
have access to buildout object, I think they can play with buildout
externals a lot, including re-running buildout (I saw in output that
buildout re-runs itself after upgrading himself).

Are extension the way to go with such thing?

If needed I am even ready to play a bit more with zc.buildout
internals code to extend if required, but according to my
understanding how buildout works there are enough places I can "hook"
to.

Any other comments? References?

Regards,
Luke