[Distutils] buildout: several fold performance increases

[Marius Gedminas]

On Sat, Jan 21, 2012 at 02:19:03AM -0800, Ross Patterson wrote:
> I moved this patch to a branch of zc.buildout:
> 
> svn+ssh://svn.zope.org/repos/main/zc.buildout/branches/env-cache
> 
[...snip bugfixing...]

> I also compared buildout run times (using the 'time' command, not
> cProfile) on a real world buildout with 6 identical parts and a few
> other parts with very similar distribution requirements.  The time
> without the patches was 1m27.513s and with the patches as applied to
> zc.buildout 1.4.4 it was 0m34.674s.

Ouch.  My slowest buildout takes "only" 41 seconds to do nothing.
Still, I'd welcome a 3x improvement.

> Also, buildout.dumppickedversions
> suffers the same logging hot spot as I fixed in zc.buildout r122980 and
> before I patched it the buildout run time was 2m13s.  Can someone cut a
> release of buildout.dumppickedversions?

There's also buildout-versions [1], which is a fork/reimplementation of
buildout.dumppickedversions, and which I use for my buildouts.  I wonder
if the same speedup could be applied to it.  Perhaps not: if I disable
buildout-versions in my 41s do-nothing buildout, the do-nothing time
stays exactly the same.

  [1] http://pypi.python.org/pypi/buildout-versions

Incidentally, looking at r122980 I wonder if you got the right revision
number [2]?  It seems to be a bugfix for LP#697913 (Buildout doesn't
honor exit code from scripts).

  [2] http://zope3.pov.lt/trac/changeset/122980

> All told, this is a 4 fold real world improvement.  With all those hot
> spots addressed there are no obvious wastes I can find in the profiling
> data.  I'd like to merge this into trunk and the 1.4 branch and see
> releases cut of both 1.5 and 1.4.  May I begin the merging?

I wish Jim Fulton would found some time to look at your branch.  4x
real-world speedup seems very compelling.

How can I test your branch?  It doesn't seem to be enough to add
 
  [buildout]
  develop += /path/to/your/branch/checkout
  [versions]
  zc.buildout = 

to my existing buildout.cfg.

Marius Gedminas
-- 
HOST SYSTEM NOT RESPONDING, PROBABLY DOWN. DO YOU WANT TO WAIT? (Y/N)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120121/3ace171e/attachment.pgp>