Any known tricks or effort made to make the PyPy source code build faster?
Hi, By default, it appears most of the time during the build/compile process, only 1 single CPU core is busy, signaling missing of parallel compiling. Is there any best known practice to make it faster? Thanks, Peter
On 30/08/2016 9:53 am, "Wang, Peter Xihong" <peter.xihong.wang@intel.com> wrote:
Hi,
By default, it appears most of the time during the build/compile process,
only 1 single CPU core is busy, signaling missing of parallel compiling. Is there any best known practice to make it faster?
Not at the moment. The STM work that Armin is doing may give us an interpreter that can make better use of extra cores during translation, but it is still experimental.
Thanks,
Peter
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
On Tue, 30 Aug 2016, William ML Leslie wrote:
On 30/08/2016 9:53 am, "Wang, Peter Xihong" <peter.xihong.wang@intel.com> wrote:
By default, it appears most of the time during the build/compile process, only 1 single CPU core is busy, signaling missing of parallel compiling. Is there any best known practice to make it faster?
Not at the moment. The STM work that Armin is doing may give us an interpreter that can make better use of extra cores during translation, but it is still experimental.
Just to elaborate on this as it might be not obvious for the newcomers: most of the "build" time is actually *not* spent upon compilation itself (as with many other projects), but rather upon translation of the interpreter into the source code, which is to be compiled into the binary during the last step of the build. This last step can be easily parallelized with make, however, because of the above, this has little practical value. Unfortunately, parallelization of the translation process is challenging due to the GIL issue. The (amazing) STM work as mentioned by William sidesteps this problem, but, at the moment, it is not ready yet. -- Sincerely yours, Yury V. Zaytsev
HI Yury, Thanks for clarifying on this. Also want to take this opportunity to thank William for the initial answer. We did notice fancy screen output during the build process to demonstrate the build progress, and I am wondering if a build option could be made to disable the output, which may save some valuable build time. Just a thought. Thanks, Peter -----Original Message----- From: Yury V. Zaytsev [mailto:yury@shurup.com] Sent: Wednesday, September 07, 2016 12:03 PM To: William ML Leslie <william.leslie.ttg@gmail.com> Cc: Wang, Peter Xihong <peter.xihong.wang@intel.com>; PyPy Developer Mailing List <pypy-dev@python.org> Subject: Re: [pypy-dev] Any known tricks or effort made to make the PyPy source code build faster? On Tue, 30 Aug 2016, William ML Leslie wrote:
On 30/08/2016 9:53 am, "Wang, Peter Xihong" <peter.xihong.wang@intel.com> wrote:
By default, it appears most of the time during the build/compile process, only 1 single CPU core is busy, signaling missing of parallel compiling. Is there any best known practice to make it faster?
Not at the moment. The STM work that Armin is doing may give us an interpreter that can make better use of extra cores during translation, but it is still experimental.
Just to elaborate on this as it might be not obvious for the newcomers: most of the "build" time is actually *not* spent upon compilation itself (as with many other projects), but rather upon translation of the interpreter into the source code, which is to be compiled into the binary during the last step of the build. This last step can be easily parallelized with make, however, because of the above, this has little practical value. Unfortunately, parallelization of the translation process is challenging due to the GIL issue. The (amazing) STM work as mentioned by William sidesteps this problem, but, at the moment, it is not ready yet. -- Sincerely yours, Yury V. Zaytsev
On Wed, 7 Sep 2016, Wang, Peter Xihong wrote:
We did notice fancy screen output during the build process to demonstrate the build progress, and I am wondering if a build option could be made to disable the output, which may save some valuable build time. Just a thought.
I understand that you are referring to the trademark Mandelbrot fractal; if that's the case, then, well, you might be able to save a few seconds on a several hours translation if you remove it, but I doubt that this would really help your cause... I would say that the most promising approach so far would be to fund and/or help with the STM work ;-) -- Sincerely yours, Yury V. Zaytsev
On 30/08/2016 9:53 am, "Wang, Peter Xihong" <peter.xihong.wang@intel.com> wrote:
By default, it appears most of the time during the build/compile process, only 1 single CPU core is busy, signaling missing of parallel compiling. Is there any best known practice to make it faster?
Now that I think about it, there is yet another not obvious "trick" that I should have probably mentioned: You should definitively be translating PyPy with PyPy itself (rather than CPython); even though it will still use only 1 core, the translation will greatly benefit from the JIT. It's been a very long while since I've last checked, but I wouldn't be surprised about a factor 2x speedup or so... -- Sincerely yours, Yury V. Zaytsev
On 07/09/16 22:07, Yury V. Zaytsev wrote:
On 30/08/2016 9:53 am, "Wang, Peter Xihong" <peter.xihong.wang@intel.com> wrote:
By default, it appears most of the time during the build/compile process, only 1 single CPU core is busy, signaling missing of parallel compiling. Is there any best known practice to make it faster?
Now that I think about it, there is yet another not obvious "trick" that I should have probably mentioned:
You should definitively be translating PyPy with PyPy itself (rather than CPython); even though it will still use only 1 core, the translation will greatly benefit from the JIT. It's been a very long while since I've last checked, but I wouldn't be surprised about a factor 2x speedup or so...
Yes, that's correct: translating pypy with itself is significantly faster. The Mandelbrot fractal takes not many seconds to compute, so is not really a problem. Cheers, Carl Friedrich
That's a great tip, :-). Just done with an experiment, and PyPy2 5.4.1-alpha0 cut down build time by 27% compared with CPython2.7.6 Thanks! Peter -----Original Message----- From: Yury V. Zaytsev [mailto:yury@shurup.com] Sent: Wednesday, September 07, 2016 1:08 PM To: Wang, Peter Xihong <peter.xihong.wang@intel.com> Cc: William ML Leslie <william.leslie.ttg@gmail.com>; PyPy Developer Mailing List <pypy-dev@python.org> Subject: RE: [pypy-dev] Any known tricks or effort made to make the PyPy source code build faster?
On 30/08/2016 9:53 am, "Wang, Peter Xihong" <peter.xihong.wang@intel.com> wrote:
By default, it appears most of the time during the build/compile process, only 1 single CPU core is busy, signaling missing of parallel compiling. Is there any best known practice to make it faster?
Now that I think about it, there is yet another not obvious "trick" that I should have probably mentioned: You should definitively be translating PyPy with PyPy itself (rather than CPython); even though it will still use only 1 core, the translation will greatly benefit from the JIT. It's been a very long while since I've last checked, but I wouldn't be surprised about a factor 2x speedup or so... -- Sincerely yours, Yury V. Zaytsev
participants (4)
-
Carl Friedrich Bolz -
Wang, Peter Xihong -
William ML Leslie -
Yury V. Zaytsev