Hey, I just translated SVN trunk of PyPy with JIT enabled and I was very pleased by the results as I noticed that PyPy with JIT is now two times faster in Pystone (I know that's not saying too much) than Cpython. However, translating took a very long time due to not being able to make use of my other CPU cores. I did not find a switch to compile using multiple shells (as in -jN) so I was wondering whether such a feature was planned or even possible. At least for the compilation part, this should be very possible. -- Sven-Hendrik
2009/12/4 Sven-Hendrik Haase <sh@lutzhaase.com>:
No, that's not planned at the moment. The only time we could easily use multiple CPU at the moment is compiling the final C sources. You can even do this manually with the make file in the temp source directory. -- Regards, Benjamin
Hi, On Fri, Dec 04, 2009 at 06:18:13PM +0100, Antonio Cuni wrote:
Of course, that is full of open problems too. The main one is that each gcc process consumes potentially a lot of RAM, so just passing "-j" is not a great idea, as all gccs are started in parallel. It looks like some obscure tweak is needed, like setting -j to a number that depends not only on the number of CPUs (as is classically done) but also on the total RAM of the system... A bientot, Armin.
On 5 Dec 2009, 04:49 pm, fijall@gmail.com wrote:
I poked around on this front a bit. I couldn't find any code in PyPy which invokes make. I did find pypy.translator.platform.distutils_platform.DistutilsPlatform._build, though. This seems to be where lists of C files are sent for compilation. Is that right? I thought about how to make this parallel. The cheesy solution, of course, would be to start a few threads and have them do the compilation (which should be sufficiently parallel, since it's another process that's doing the actual work). This is a bit complicated by the chdir calls in the code, though. Also, maybe distutils isn't threadsafe. I dunno if I'll think about this any further, but I thought I'd summarize what little I did figure out. Jean-Paul
Hello, 2010/1/9 <exarkun@twistedmatrix.com>:
PyPy does generate a makefile (gen_makefile() in http://codespeak.net/svn/pypy/trunk/pypy/translator/c/genc.py ) but it is not used in all configurations: In the same file, see the call to execute_makefile(). -Ojit uses the makefile, though. -- Amaury Forgeot d'Arc
On Sat, Dec 5, 2009 at 16:44, Armin Rigo <arigo@tunes.org> wrote:
My 2 cents: for C++ I would be really worried, but for C I'd guess that even with 4 CPUs (i.e. classically 5 jobs) one is not going to go over 1Gb... or not? Dear Sven-Hendrik, would you verify this idea by launching make by hand, as suggested? (Now, I'm going back to just lurking). Regards -- Paolo Giarrusso
Armin Rigo, 05.12.2009 16:44:
I just did a quick check with lxml.etree, for which Cython generates a 6.5MB C file with 150K lines (~96K non-empty/non-'#' lines in gcc -E). Running that through "gcc -O3 -march=core2 -pipe" keeps the peek virtual memory allocation in 'top' well below 350MB on my 32bit Linux system. Developer machines tend to be rather well equipped these days, so not much to worry about here, IMHO. Stefan
2009/12/4 Sven-Hendrik Haase <sh@lutzhaase.com>:
No, that's not planned at the moment. The only time we could easily use multiple CPU at the moment is compiling the final C sources. You can even do this manually with the make file in the temp source directory. -- Regards, Benjamin
Hi, On Fri, Dec 04, 2009 at 06:18:13PM +0100, Antonio Cuni wrote:
Of course, that is full of open problems too. The main one is that each gcc process consumes potentially a lot of RAM, so just passing "-j" is not a great idea, as all gccs are started in parallel. It looks like some obscure tweak is needed, like setting -j to a number that depends not only on the number of CPUs (as is classically done) but also on the total RAM of the system... A bientot, Armin.
On 5 Dec 2009, 04:49 pm, fijall@gmail.com wrote:
I poked around on this front a bit. I couldn't find any code in PyPy which invokes make. I did find pypy.translator.platform.distutils_platform.DistutilsPlatform._build, though. This seems to be where lists of C files are sent for compilation. Is that right? I thought about how to make this parallel. The cheesy solution, of course, would be to start a few threads and have them do the compilation (which should be sufficiently parallel, since it's another process that's doing the actual work). This is a bit complicated by the chdir calls in the code, though. Also, maybe distutils isn't threadsafe. I dunno if I'll think about this any further, but I thought I'd summarize what little I did figure out. Jean-Paul
Hello, 2010/1/9 <exarkun@twistedmatrix.com>:
PyPy does generate a makefile (gen_makefile() in http://codespeak.net/svn/pypy/trunk/pypy/translator/c/genc.py ) but it is not used in all configurations: In the same file, see the call to execute_makefile(). -Ojit uses the makefile, though. -- Amaury Forgeot d'Arc
On Sat, Dec 5, 2009 at 16:44, Armin Rigo <arigo@tunes.org> wrote:
My 2 cents: for C++ I would be really worried, but for C I'd guess that even with 4 CPUs (i.e. classically 5 jobs) one is not going to go over 1Gb... or not? Dear Sven-Hendrik, would you verify this idea by launching make by hand, as suggested? (Now, I'm going back to just lurking). Regards -- Paolo Giarrusso
Armin Rigo, 05.12.2009 16:44:
I just did a quick check with lxml.etree, for which Cython generates a 6.5MB C file with 150K lines (~96K non-empty/non-'#' lines in gcc -E). Running that through "gcc -O3 -march=core2 -pipe" keeps the peek virtual memory allocation in 'top' well below 350MB on my 32bit Linux system. Developer machines tend to be rather well equipped these days, so not much to worry about here, IMHO. Stefan
participants (9)
-
Amaury Forgeot d'Arc -
Antonio Cuni -
Armin Rigo -
Benjamin Peterson -
exarkun＠twistedmatrix.com -
Maciej Fijalkowski -
Paolo Giarrusso -
Stefan Behnel -
Sven-Hendrik Haase