Translating PyPy on Windows
Hi everyone! When I translated PyPy on Windows, on a laptop with an Intel 3210M processor, I got the following timings: annotate --- 464.5 s rtype_lltype --- 882.7 s pyjitpl_lltype --- 825.9 s backendopt_lltype --- 259.2 s stackcheckinsertion_lltype --- 309.0 s database_c --- 349.6 s source_c --- 502.4 s compile_c --- 3830.5 s =========================================== Total: --- 7423.8 s More than half the time is spent compiling C code. The makefile uses neither precompiled headers nor parallel compilation, so there is lots of room for improvement. I tried these two optimizations, and here are the timings for the C code compilation: baseline: 3830.5 s (as above) pc: 1301.2 s pch: 404.8 s pch + pc: 165.8 s pc = parallel compilation pch = precompiled headers Using both optimizations you get a factor 23 speed up! Adding parallel compilation is easy. Adding precompiled headers is a bit more work; here I did a quick fix that is not suitable for production code, but good enough for profiling. I plan to add parallel compilation on Windows, along with some other Windows related fixes, to the PyPy code. I will not add precompiled headers now. Armin, Maciej: Does that sound OK? ---------------------------------------------------------- The main gain when using precompiled headers comes from the header forwarddecl.h. This header is 17.8 MB and is included in 635 source files. The amount of C code that is compiled is about 424 MB. But because of forwarddecl.h, you are in fact compiling another 635 x 17.8 MB = 11 GB. Compiling 424 MB or 11 GB of C code - that does make a difference. But you can not just create a precompiled header that includes forwarddecl.h and apply it to all source files. You should only apply the precompiled header to files that include forwarddecl.h to begin with. These seem to be precisely the source files with names that match one of the patterns data_*.c, nonfuncnodes*.c, implement*.c, pypy_*.c and rpython_*.c. But I don't see how to write a nmake rule that only applies to such files. (nmake does not have as good pattern matching support as GNU make.) --Johan
On Sun, Jan 26, 2014 at 1:10 PM, Johan Råde <johan.rade@gmail.com> wrote:
Hi everyone!
When I translated PyPy on Windows, on a laptop with an Intel 3210M processor, I got the following timings:
annotate --- 464.5 s rtype_lltype --- 882.7 s pyjitpl_lltype --- 825.9 s backendopt_lltype --- 259.2 s stackcheckinsertion_lltype --- 309.0 s database_c --- 349.6 s source_c --- 502.4 s compile_c --- 3830.5 s =========================================== Total: --- 7423.8 s
More than half the time is spent compiling C code. The makefile uses neither precompiled headers nor parallel compilation, so there is lots of room for improvement. I tried these two optimizations, and here are the timings for the C code compilation:
baseline: 3830.5 s (as above) pc: 1301.2 s pch: 404.8 s pch + pc: 165.8 s
pc = parallel compilation pch = precompiled headers
Using both optimizations you get a factor 23 speed up!
Adding parallel compilation is easy.
Adding precompiled headers is a bit more work; here I did a quick fix that is not suitable for production code, but good enough for profiling.
I plan to add parallel compilation on Windows, along with some other Windows related fixes, to the PyPy code. I will not add precompiled headers now.
Armin, Maciej: Does that sound OK?
----------------------------------------------------------
The main gain when using precompiled headers comes from the header forwarddecl.h. This header is 17.8 MB and is included in 635 source files. The amount of C code that is compiled is about 424 MB. But because of forwarddecl.h, you are in fact compiling another 635 x 17.8 MB = 11 GB. Compiling 424 MB or 11 GB of C code - that does make a difference.
But you can not just create a precompiled header that includes forwarddecl.h and apply it to all source files. You should only apply the precompiled header to files that include forwarddecl.h to begin with. These seem to be precisely the source files with names that match one of the patterns data_*.c, nonfuncnodes*.c, implement*.c, pypy_*.c and rpython_*.c. But I don't see how to write a nmake rule that only applies to such files. (nmake does not have as good pattern matching support as GNU make.)
--Johan
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Hi Johan Should this mail come with some attachments? Cheers, fijal
On 2014-01-26 13:34, Maciej Fijalkowski wrote:
Hi Johan
Should this mail come with some attachments?
Cheers, fijal _______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Hi Maciej, No attachments. I haven't written the actual Python code yet, I just hacked the generated makefile by hand. Cheers, Johan
On Sun, Jan 26, 2014 at 2:44 PM, Johan Råde <johan.rade@gmail.com> wrote:
On 2014-01-26 13:34, Maciej Fijalkowski wrote:
Hi Johan
Should this mail come with some attachments?
Cheers, fijal _______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Hi Maciej,
No attachments. I haven't written the actual Python code yet, I just hacked the generated makefile by hand.
Cheers,
Johan
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
If you wish, post the hacked Makefile to a pastebin and we can work backwards from there. I would be willing to help move this forward. Matti
On 2014-01-26 15:15, matti picus wrote:
On Sun, Jan 26, 2014 at 2:44 PM, Johan Råde <johan.rade@gmail.com
If you wish, post the hacked Makefile to a pastebin and we can work backwards from there. I would be willing to help move this forward. Matti
Hi Matti, Here it is https://dl.dropboxusercontent.com/u/525329/Makefile It is the first makefile I write ever. You have been warned. --Johan
Hi Matti, Two weeks ago I promised I would take care of orecompiled headers in two weeks. That's now. But it seems you already did it. I tested your code on Windows with MSVS, and it works well. How much did you gain on Linux with GCC? --Johan On 2014-01-26 15:15, matti picus wrote:
If you wish, post the hacked Makefile to a pastebin and we can work backwards from there. I would be willing to help move this forward. Matti
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
Hi Maciej, Here are the details of what I did when I tried using precompiled headers. 1. I checked out revision 68823 2. I translated PyPy. 3. I went to the directory with all (well most) of the sources. On my machine it was C:\Users\Rade\AppData\Local\Temp\usession-default-0\testing_1 4. I added include guards to common_header.h and preimpl.h 5. I added a file stdafx.c with the single line #include "stdafx.h" 6. I added a file stdafx.h with the the lines #ifndef PYPY_STDAFX_H #define PYPY_STDAFX_H #include "common_header.h" #include "structdef.h" #include "forwarddecl.h" #include "preimpl.h" #endif 7. I revised the makefile so that it would created a precompiled header from stdafx.c and compile all the other source files with that precompiled header. The commands were essentially cl.exe stdafx.c $(CFLAGS) /Ycstdafx.h /Fpstdafx.pch $(INCLUDEDIRS) cl.exe <filename> $(CFLAGS) /Yustdafx.h /Fpstdafx.pch /FIstdafx.h $(INCLUDEDIRS) Here /Yc = create precompiled header /Yu = use precompiled header /Fp = specify precompiled header name /FI = force inclusion of header file Things would of course be different with gcc. Then I ran the makefile. Several files did not compile. 8. I revised the makefile so that the source files that did not compile in step 7 were compiled without the precompiled header. I ran the makefile again. It compiled and linked and the generated executable seemed to work. ---------------------------------- What should be done in addition: 1. The precompiled header should only be used with files that begin with #include "common_header.h" #include "structdef.h" #include "forwarddecl.h" #include "preimpl.h" That seems to be the files with names that match one of the patterns data_*.h nonfuncnodes*.h implement*.h pypy_*.h rpython_*.h The remaining files either include python.h or or just plain standard C lib headers. 2. The source files typically begin with something like #define PYPY_FILE_NAME "rpython_rtyper_lltypesystem_rlist.c" #include "common_header.h" #include "structdef.h" #include "forwarddecl.h" #include "preimpl.h" #include "src/g_include.h" The first line is different in each file. This seems to indicate that you can not use precompiled headers. However, the macro PYPY_FILE_NAME is only used in src/g_include.h. Thus the files could, and should, be rewritten as #include "common_header.h" #include "structdef.h" #include "forwarddecl.h" #include "preimpl.h" #define PYPY_FILE_NAME "rpython_rtyper_lltypesystem_rlist.c" #include "src/g_include.h" Now it becomes clear that you can put the first four lines in a precompiled header. 3. Optionally one could refactor src/g_include.h so it does not rely on the macro PYPY_FILE_NAME. Then you could add src/g_include.h to the precompiled header as well. But I think the performance gain from that would be small. Cheers, Johan
participants (4)
-
Johan Råde -
Maciej Fijalkowski -
matti picus -
Matti Picus