[Distutils] buildout: several fold performance increases
marius at pov.lt
Sat Jan 21 16:00:16 CET 2012
On Sat, Jan 21, 2012 at 02:19:03AM -0800, Ross Patterson wrote:
> I moved this patch to a branch of zc.buildout:
> I also compared buildout run times (using the 'time' command, not
> cProfile) on a real world buildout with 6 identical parts and a few
> other parts with very similar distribution requirements. The time
> without the patches was 1m27.513s and with the patches as applied to
> zc.buildout 1.4.4 it was 0m34.674s.
Ouch. My slowest buildout takes "only" 41 seconds to do nothing.
Still, I'd welcome a 3x improvement.
> Also, buildout.dumppickedversions
> suffers the same logging hot spot as I fixed in zc.buildout r122980 and
> before I patched it the buildout run time was 2m13s. Can someone cut a
> release of buildout.dumppickedversions?
There's also buildout-versions , which is a fork/reimplementation of
buildout.dumppickedversions, and which I use for my buildouts. I wonder
if the same speedup could be applied to it. Perhaps not: if I disable
buildout-versions in my 41s do-nothing buildout, the do-nothing time
stays exactly the same.
Incidentally, looking at r122980 I wonder if you got the right revision
number ? It seems to be a bugfix for LP#697913 (Buildout doesn't
honor exit code from scripts).
> All told, this is a 4 fold real world improvement. With all those hot
> spots addressed there are no obvious wastes I can find in the profiling
> data. I'd like to merge this into trunk and the 1.4 branch and see
> releases cut of both 1.5 and 1.4. May I begin the merging?
I wish Jim Fulton would found some time to look at your branch. 4x
real-world speedup seems very compelling.
How can I test your branch? It doesn't seem to be enough to add
develop += /path/to/your/branch/checkout
to my existing buildout.cfg.
HOST SYSTEM NOT RESPONDING, PROBABLY DOWN. DO YOU WANT TO WAIT? (Y/N)
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 190 bytes
Desc: Digital signature
More information about the Distutils-SIG