[Distutils] buildout: several fold performance increases

[Ross Patterson]

Marius Gedminas <marius at pov.lt> writes:

> On Sat, Jan 21, 2012 at 02:19:03AM -0800, Ross Patterson wrote:
>> I moved this patch to a branch of zc.buildout:
>> 
>> svn+ssh://svn.zope.org/repos/main/zc.buildout/branches/env-cache

[...snip...]

>> Also, buildout.dumppickedversions
>> suffers the same logging hot spot as I fixed in zc.buildout r122980 and
>> before I patched it the buildout run time was 2m13s.  Can someone cut a
>> release of buildout.dumppickedversions?
>
> There's also buildout-versions [1], which is a fork/reimplementation of
> buildout.dumppickedversions, and which I use for my buildouts.  I wonder
> if the same speedup could be applied to it.  Perhaps not: if I disable
> buildout-versions in my 41s do-nothing buildout, the do-nothing time
> stays exactly the same.
>
>   [1] http://pypi.python.org/pypi/buildout-versions

Yup, it does suffer the same:

https://github.com/Simplistix/buildout-versions/blob/master/src/buildout_versions/__init__.py#L20

That speedup is proportional to the number of required dists, so your
do-nothing might not suffer much.  In fact, I think all of these
optimizations are at least proportional to the size of your egg cache,
IOW, the number of possible dists available on your path.

> Incidentally, looking at r122980 I wonder if you got the right revision
> number [2]?  It seems to be a bugfix for LP#697913 (Buildout doesn't
> honor exit code from scripts).
>
>   [2] http://zope3.pov.lt/trac/changeset/122980

Oops, sorry about that, it's r124059.  Don't know how I got that wrong:

http://zope3.pov.lt/trac/changeset/124059

> How can I test your branch?  It doesn't seem to be enough to add
>  
>   [buildout]
>   develop += /path/to/your/branch/checkout
>   [versions]
>   zc.buildout = 
>
> to my existing buildout.cfg.

Not sure how.  I've been patching eggs to do the 'timed' tests with
real-world buildout.  Maybe you could install the checkout as a develop
egg to the python installation?  Ick.  Maybe you could edit bin/buildout
and adjust the path for the buildout egg?  Ick.

Ross