Struggling with a modular buildout configuration
Hi all, I'm trying to setup a modular buildout configuration, but I'm hitting my head on an invisible wall... Basically, I'd like to have a top-level buildout.cfg that "collects" settings from various "subprojects"; in particular, each subproject may "augment" a set of eggs, some of which may be defined as "develop" (in buildout parlance). Finally, with a single "zc.recipe.egg" I wanna install a custom interpreter with all those eggs inside. So, I have the top-level configuration that uses "extends" to include a "projects.cfg": [buildout] extends = base.cfg projects.cfg that in turn brings in the various subprojects snippets thru "extends": [buildout] extends = src/this-project.cfg src/that-project.cfg these snippets typically contain a section like [projects] eggs += this-project-egg some-other-dependency-egg develop += src/this-project parts += this-project-specific-part [this-project-specific-part] ... The problem I'm having is that for incomprensible (to me, of course :) reason some of those settings do not "emerge" at the top level (others do!), as if a later "extends" is clobbering a previous one. I built a minimalistic example [1] that shows the unexpected behaviour; doing a "buildout annotate" I get something like the following: Annotated sections ================== [buildout] ... develop= ${projects:develop} /tmp/tb/buildout.cfg parts= ${projects:parts} /tmp/tb/buildout.cfg ... [extra-parts-from-proj2] [projects] develop= src/proj2 += /tmp/tb/src/proj2/devel.cfg eggs= extra-needed-by-proj1 proj1 proj2 += /tmp/tb/src/proj1/devel.cfg += /tmp/tb/src/proj1/base.cfg += /tmp/tb/src/proj2/base.cfg parts= extra-parts-from-proj2 += /tmp/tb/src/proj2/devel.cfg [python] eggs= ${projects:eggs} /tmp/tb/base.cfg interpreter= python /tmp/tb/base.cfg recipe= zc.recipe.egg /tmp/tb/base.cfg If you look at the [projects] section, you can see that while the "eggs" setting carries the three expected values (one for each subproject plus one extra), the "develop" setting contains just a single value, the one injected by proj2, and the one that should be coming from proj1 is missing. If I omit "src/proj2" from the extends option, then the "develop" setting correctly contains just "src/proj1"... Of course I tried hard to understand what's going, even walking into buildout 1.5.2 code with pdb. I tried also with the latest development version (the upcoming 1.6) as well as the github branch (IIUC it derivates from 1.4), but all exhibit the same behaviour. The 1.6 code is slightly different, and at first it seemed doing the right thing, but at a closer look it really moved the problem from the "develop" setting "eggs". As far as I can tell, there is something strange happening in the _update_section() function when the _open() function handles the "extends" setting: at some point it "loses" the cognition that a certain assignment is an expansion (tecnically, that the key is actually "develop +", instead of "develop"), so the final "s1.update(s2)" overrides it, instead of augmenting it... Of course I may be doing the wrong thing or missing the obvious, and if so I'll be happy if you can shed some light on me. Thanks in advance for any help and sorry for the long post, ciao, lele. [1] http://artiemestieri.tn.it/~lele/test-buildout.tar.gz -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele@metapensiero.it | -- Fortunato Depero, 1929.
Hi Lele, can you provide an example that would illustrate the problem? The file you submitted fails because projects do not include setup.py. I would advise to use zc.buildout 1.6.0 and see if issue persists.
can you provide an example that would illustrate the problem? The file you submitted fails because projects do not include setup.py. I would advise to use zc.buildout 1.6.0 and see if issue persists.
Sure, I added minimalistic setup.py to both subprojects: http://artiemestieri.tn.it/~lele/test-buildout-2.tar.gz As is, executing a buildout (btw, using latest 1.6.1 now), I get: $ buildout Creating directory '/tmp/test-zcb/bin'. Creating directory '/tmp/test-zcb/parts'. Creating directory '/tmp/test-zcb/develop-eggs'. Develop: '/tmp/test-zcb/src/proj2' Installing extra-parts-from-proj2. Note it does not even mention anything from proj1, even if it is surely loading its configuration snippets: $ buildout annotate | grep proj1 proj1 += /tmp/test-zcb/src/proj1/devel.cfg += /tmp/test-zcb/src/proj1/base.cfg and it does not install the [python] recipe neither... If I comment out the last line in the top level projects.cfg, the one that includes the proj2/devel.cfg, I get: $ buildout Develop: '/tmp/test-zcb/src/proj1' Running uninstall recipe. Installing python. Generated interpreter '/tmp/test-zcb/bin/python'. This time it installed the [python] recipe, and effectively $ bin/python -m project1 Hello from project1 with the extra package too: $ bin/python -c "import faker; print faker.__version__" 0.0.4 So, it really seems that loading the projX extension clobbers the previous settings, but I cannot see what I'm doing wrong. Thanks a lot for your time, and for any enlightenment :-) ciao, lele. PS: I won't be able to follow the group for the next two weeks as I'm taking a long awaited vacation.
Lele Gaifax <lele@metapensiero.it> writes:
can you provide an example that would illustrate the problem? The file you submitted fails because projects do not include setup.py. I would advise to use zc.buildout 1.6.0 and see if issue persists.
Sure, I added minimalistic setup.py to both subprojects:
Hi Domen, just to shake a doubt, was that enough for you to exercise the problem, or can/should I provide more details? Thank you, ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele@metapensiero.it | -- Fortunato Depero, 1929.
On 2012-09-05 09:55:52 +0000, Lele Gaifax said:
Lele Gaifax <lele@metapensiero.it> writes:
can you provide an example that would illustrate the problem? The file you submitted fails because projects do not include setup.py. I would advise to use zc.buildout 1.6.0 and see if issue persists.
Sure, I added minimalistic setup.py to both subprojects:
Hi Domen,
just to shake a doubt, was that enough for you to exercise the problem, or can/should I provide more details?
Did zc.buildout 1.6.3 help at all?
Thank you, ciao, lele.
-- Alex Clark · http://pythonpackages.com
Alex Clark <aclark@aclark.net> writes:
Did zc.buildout 1.6.3 help at all?
No, it exhibits the very same behaviour as described in my previous post, that is, the configuration bits from proj2 clobber those coming from proj1. While it's enterely possible I'm misusing the extends functionality, it seems something odd is happening in the _update_section() function... Thanks and bye, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele@metapensiero.it | -- Fortunato Depero, 1929.
Hi, I just pushed an even simpler test case that shows the "defect": https://github.com/lelit/buildout/commit/1dc3d6af0b6326500ad72798352dd153736... that hopefully will help spotting the bug (or maybe my bad usage of the extends functionality :-) I already tried to tweak the _update_section() function, but without success: AFAIU, with the test above at some point the destination section (`s1` in the function) contains *both* a "egg" and a "egg +" keys, so the line # Find v1 in s2 first; it may have been defined locally too. v1, note1 = s2.get(key, s1.get(key, ("", ""))) where `key` is "egg" ignores the s1['egg +'] value... I'll try to spend some more hours on it. Thanks and bye, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele@metapensiero.it | -- Fortunato Depero, 1929.
participants (3)
-
Alex Clark
-
Domen Kožar
-
Lele Gaifax