[Distutils] Struggling with a modular buildout configuration

Lele Gaifax lele at metapensiero.it
Wed Aug 15 16:36:56 CEST 2012

Hi all,

I'm trying to setup a modular buildout configuration, but I'm hitting my
head on an invisible wall...

Basically, I'd like to have a top-level buildout.cfg that "collects"
settings from various "subprojects"; in particular, each subproject may
"augment" a set of eggs, some of which may be defined as "develop" (in
buildout parlance). Finally, with a single "zc.recipe.egg" I wanna
install a custom interpreter with all those eggs inside.

So, I have the top-level configuration that uses "extends" to include a

  extends = base.cfg projects.cfg
that in turn brings in the various subprojects snippets thru "extends":

  extends = src/this-project.cfg src/that-project.cfg

these snippets typically contain a section like

  eggs += this-project-egg some-other-dependency-egg
  develop += src/this-project
  parts += this-project-specific-part


The problem I'm having is that for incomprensible (to me, of course :)
reason some of those settings do not "emerge" at the top level (others
do!), as if a later "extends" is clobbering a previous one.

I built a minimalistic example [1] that shows the unexpected behaviour;
doing a "buildout annotate" I get something like the following:

  Annotated sections

  develop= ${projects:develop}
  parts= ${projects:parts}


  +=  /tmp/tb/src/proj2/devel.cfg
  +=  /tmp/tb/src/proj1/devel.cfg
  +=  /tmp/tb/src/proj1/base.cfg
  +=  /tmp/tb/src/proj2/base.cfg
  +=  /tmp/tb/src/proj2/devel.cfg

  eggs= ${projects:eggs}
  interpreter= python
  recipe= zc.recipe.egg

If you look at the [projects] section, you can see that while the "eggs"
setting carries the three expected values (one for each subproject plus
one extra), the "develop" setting contains just a single value, the one
injected by proj2, and the one that should be coming from proj1 is
missing. If I omit "src/proj2" from the extends option, then the
"develop" setting correctly contains just "src/proj1"...

Of course I tried hard to understand what's going, even walking into
buildout 1.5.2 code with pdb. I tried also with the latest development
version (the upcoming 1.6) as well as the github branch (IIUC it
derivates from 1.4), but all exhibit the same behaviour. The 1.6 code is
slightly different, and at first it seemed doing the right thing, but at
a closer look it really moved the problem from the "develop" setting

As far as I can tell, there is something strange happening in the
_update_section() function when the _open() function handles the
"extends" setting: at some point it "loses" the cognition that a certain
assignment is an expansion (tecnically, that the key is actually
"develop +", instead of "develop"), so the final "s1.update(s2)"
overrides it, instead of augmenting it...

Of course I may be doing the wrong thing or missing the obvious, and if
so I'll be happy if you can shed some light on me.

Thanks in advance for any help and sorry for the long post,

ciao, lele.

[1] http://artiemestieri.tn.it/~lele/test-buildout.tar.gz
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
lele at metapensiero.it  |                 -- Fortunato Depero, 1929.

More information about the Distutils-SIG mailing list