[Distutils] Struggling with a modular buildout configuration

[Lele Gaifax]

Hi all,

I'm trying to setup a modular buildout configuration, but I'm hitting my
head on an invisible wall...

Basically, I'd like to have a top-level buildout.cfg that "collects"
settings from various "subprojects"; in particular, each subproject may
"augment" a set of eggs, some of which may be defined as "develop" (in
buildout parlance). Finally, with a single "zc.recipe.egg" I wanna
install a custom interpreter with all those eggs inside.

So, I have the top-level configuration that uses "extends" to include a
"projects.cfg":

  [buildout]
  extends = base.cfg projects.cfg
  
that in turn brings in the various subprojects snippets thru "extends":

  [buildout]
  extends = src/this-project.cfg src/that-project.cfg

these snippets typically contain a section like

  [projects]
  eggs += this-project-egg some-other-dependency-egg
  develop += src/this-project
  parts += this-project-specific-part

  [this-project-specific-part]
  ...

The problem I'm having is that for incomprensible (to me, of course :)
reason some of those settings do not "emerge" at the top level (others
do!), as if a later "extends" is clobbering a previous one.

I built a minimalistic example [1] that shows the unexpected behaviour;
doing a "buildout annotate" I get something like the following:

  Annotated sections
  ==================

  [buildout]
  ...
  develop= ${projects:develop}
      /tmp/tb/buildout.cfg
  parts= ${projects:parts}
      /tmp/tb/buildout.cfg
  ...

  [extra-parts-from-proj2]

  [projects]
  develop= 
  src/proj2
  +=  /tmp/tb/src/proj2/devel.cfg
  eggs= 
  extra-needed-by-proj1
  proj1
  proj2
  +=  /tmp/tb/src/proj1/devel.cfg
  +=  /tmp/tb/src/proj1/base.cfg
  +=  /tmp/tb/src/proj2/base.cfg
  parts= 
  extra-parts-from-proj2
  +=  /tmp/tb/src/proj2/devel.cfg

  [python]
  eggs= ${projects:eggs}
      /tmp/tb/base.cfg
  interpreter= python
      /tmp/tb/base.cfg
  recipe= zc.recipe.egg
      /tmp/tb/base.cfg

If you look at the [projects] section, you can see that while the "eggs"
setting carries the three expected values (one for each subproject plus
one extra), the "develop" setting contains just a single value, the one
injected by proj2, and the one that should be coming from proj1 is
missing. If I omit "src/proj2" from the extends option, then the
"develop" setting correctly contains just "src/proj1"...

Of course I tried hard to understand what's going, even walking into
buildout 1.5.2 code with pdb. I tried also with the latest development
version (the upcoming 1.6) as well as the github branch (IIUC it
derivates from 1.4), but all exhibit the same behaviour. The 1.6 code is
slightly different, and at first it seemed doing the right thing, but at
a closer look it really moved the problem from the "develop" setting
"eggs".

As far as I can tell, there is something strange happening in the
_update_section() function when the _open() function handles the
"extends" setting: at some point it "loses" the cognition that a certain
assignment is an expansion (tecnically, that the key is actually
"develop +", instead of "develop"), so the final "s1.update(s2)"
overrides it, instead of augmenting it...

Of course I may be doing the wrong thing or missing the obvious, and if
so I'll be happy if you can shed some light on me.

Thanks in advance for any help and sorry for the long post,

ciao, lele.

[1] http://artiemestieri.tn.it/~lele/test-buildout.tar.gz
-- 
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
lele at metapensiero.it  |                 -- Fortunato Depero, 1929.