[pypy-dev] Parallel translation?

Stefan Behnel stefan_ml at behnel.de
Mon Dec 7 10:48:19 CET 2009


Armin Rigo, 05.12.2009 16:44:
> On Fri, Dec 04, 2009 at 06:18:13PM +0100, Antonio Cuni wrote:
>> I agree that at this point in time we cannot or don't want to make 
>> annotation/rtyping/backend parallelizable, but it should definitely be 
>> possible to just pass the -j flag to 'make' in an automatic way.
> 
> Of course, that is full of open problems too.  The main one is that each
> gcc process consumes potentially a lot of RAM, so just passing "-j" is
> not a great idea, as all gccs are started in parallel.  It looks like
> some obscure tweak is needed, like setting -j to a number that depends
> not only on the number of CPUs (as is classically done) but also on the
> total RAM of the system...

I just did a quick check with lxml.etree, for which Cython generates a
6.5MB C file with 150K lines (~96K non-empty/non-'#' lines in gcc -E).
Running that through "gcc -O3 -march=core2 -pipe" keeps the peek virtual
memory allocation in 'top' well below 350MB on my 32bit Linux system.
Developer machines tend to be rather well equipped these days, so not much
to worry about here, IMHO.

Stefan




More information about the Pypy-dev mailing list