[Distutils] buildout: several fold performance increases

[Marius Gedminas]

On Thu, Jan 26, 2012 at 01:48:05AM +0100, Maurits van Rees wrote:
> Op 21-01-12 20:05, Ross Patterson schreef:
> >That speedup is proportional to the number of required dists, so your
> >do-nothing might not suffer much.  In fact, I think all of these
> >optimizations are at least proportional to the size of your egg cache,
> >IOW, the number of possible dists available on your path.
> 
> I had over 5000 eggs in an eggs directory that I share over lots of
> buildouts.  I have now removed the oldest 4000 eggs.  Then I reran
> buildout for a few projects; this added several of those eggs again
> as they were still used after all.  Then I compared the time taken
> by a buildout before and after.
> 
> A sample output of old data was this:
> real    0m59.552s
> user    0m29.117s
> sys     0m4.197s
> 
> and new data was this:
> real	0m49.364s
> user	0m19.654s
> sys	0m3.578s
> 
> So that is about a 30% time decrease; I have also seen about 50% decrease.
> 
> So removing unused eggs indeed helps.  That is good to know, thanks.

Some info about my timing data:

  $ ls ~/.buildout/eggs/|wc -l
  1791

And since I suspect there's some filename-based filtering for Python version
numbers:

  $ ls -d ~/.buildout/eggs/*-py2.6.egg|wc -l
  703

Marius Gedminas
-- 
It's my understanding that although in principle TCP can handle huge
throughputs in practice many stacks haven't been optimized for that case, so
you have to either use a utility which opens multiple TCP sessions in parallel
or do something really radical like upgrade to the latest version of the linux
kernel.
		-- Bram Cohen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20120126/1ff22c93/attachment.pgp>