[Distutils] buildout: several fold performance increases

[Marius Gedminas]

On Mon, Jan 16, 2012 at 06:45:14PM -0800, Ross Patterson wrote:
> I've long been perplexed by how long a buildout takes to run with
> multiple parts whose required distributions are largely similar.  Taking
> a stab at it, I found two hot spots that yield several fold improvements
> in performance.

That is awesome!  I spend way too much time waiting for bin/buildout.

> First, zc.buildout.easy_install._log_requirements was doing expensive
> requirements parsing and sorting even when no message would be logged.
> I committed a fix for it that on a 10 part buildout with a large "eggs"
> option for each part decreased update time from a cProfile run time of
> 93 seconds to 15 seconds:
> 
> http://svn.zope.org/zc.buildout/trunk/src/zc/buildout/easy_install.py?rev=124059&r1=122980&r2=124059

cProfiler overhead is rarely linear.  What's the real speedup when run
without a profiler?

> Unfortunately, I haven't been able to get a clean test environment for
> the life of me.  I'm using a clean Python 2.7 build from source, turning
> everything in ~/.buildout/default.cfg off, and running tests in a clean
> checkout of the zc.buildout/trunk buildout.  Even under those conditions
> I get 17 failing tests before any changes.  With this environments
> cache, I see 41 failures, but I can't make sense of it.  This patch
> yields another 2-3 fold decrease to 6 seconds for the same buildout and
> is driven by profiling data, not guessing.  Can someone help me get this
> patch in?

I'd love to try, but I'm not one of the people who understand buildout
internals (or even are able to get a clean test run on trunk).

> Finally, it would be great to see releases of zc.buildout with these
> performance improvements get out in the world.  I've been hearing more
> and more complaints about buildout run times and these are easy fixes.
> If we can get the second, attached patch in quickly, then I'd say we
> should release with both.  If not, then it's still worth it to cut a
> release for the first, already committed patch, which yields the
> greatest improvement.

Marius Gedminas
-- 
We've found by experience that people who are careless and sloppy writers are
usually also careless and sloppy thinkers (often enough to bet on, anyway).
	-- ESR (http://www.tuxedo.org/~esr/faqs/smart-questions.html)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120117/f1520daf/attachment.pgp>