Hi Maciej, This PGO is done for the interpreted code not the JITed code. Yash would give more details on this. Part of reason prompted us to do this work is due to 80% of the CPU cycles being spent in interpreted mode, while running OpenStack Swift in a real world scaled cluster setup. We've got 2% gain from this PGO effort on OpenStack, which is not bad at all. Thanks, Peter   -----Original Message----- From: pypy-dev [mailto:pypy-dev-bounces+peter.xihong.wang=intel.com@python.org] On Behalf Of pypy-dev-request@python.org Sent: Friday, November 11, 2016 9:00 AM To: pypy-dev@python.org Subject: pypy-dev Digest, Vol 67, Issue 9 Send pypy-dev mailing list submissions to pypy-dev@python.org To subscribe or unsubscribe via the World Wide Web, visit https://mail.python.org/mailman/listinfo/pypy-dev or, via email, send a message with subject or body 'help' to pypy-dev-request@python.org You can reach the person managing the list at pypy-dev-owner@python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of pypy-dev digest..." Today's Topics: 1. Re: PGO Optimized Binary (Maciej Fijalkowski) 2. speed.pypy.org and old versions (Matti Picus) 3. release tarballs of the latest PyPy2.7 available on bitbucket (Matti Picus) 4. Re: speed.pypy.org and old versions (Carl Friedrich Bolz) ---------------------------------------------------------------------- Message: 1 Date: Thu, 10 Nov 2016 20:42:37 +0200 From: Maciej Fijalkowski <fijall@gmail.com> To: "Singh, Yashwardhan" <yashwardhan.singh@intel.com> Cc: Armin Rigo <armin.rigo@gmail.com>, "pypy-dev@python.org" <pypy-dev@python.org> Subject: Re: [pypy-dev] PGO Optimized Binary Message-ID: <CAK5idxTd841A3r=Q6ZjzF8e5AGYqrzMa8Oyqwep3uF0j4zA1qQ@mail.gmail.com> Content-Type: text/plain; charset=UTF-8 Hi 8% of that is very good if you can reproduce it across multiple runs (there is a pretty high variance I would think). You can also try running with --jit off. This gives you an indication of the speed of interpreter, which is a part of warmup On Wed, Nov 9, 2016 at 12:30 AM, Singh, Yashwardhan <yashwardhan.singh@intel.com> wrote:

Hi Armin,

Thanks for your feedback. We ran one of the program suggested by you as an example for evaluation: cd rpython/jit/tl non-pgo-pypy ../../bin/rpython -O2 --source targettlr pgo-pypy ../../bin/rpython -O2 --source targettlr

We got the following results : Non-Pgo pypy - [Timer] Timings: [Timer] annotate --- 7.5 s [Timer] rtype_lltype --- 5.8 s [Timer] backendopt_lltype --- 3.6 s [Timer] stackcheckinsertion_lltype --- 0.1 s [Timer] database_c --- 19.6 s [Timer] source_c --- 2.6 s [Timer] ========================================= [Timer] Total: --- 39.2 s

PGO-pypy : [Timer] Timings: [Timer] annotate --- 7.6 s [Timer] rtype_lltype --- 5.1 s [Timer] backendopt_lltype --- 3.1 s [Timer] stackcheckinsertion_lltype --- 0.0 s [Timer] database_c --- 18.5 s [Timer] source_c --- 2.3 s [Timer] ========================================= [Timer] Total: --- 36.6 s

The delta in performance between these two is about 8%.

We are working on getting the data to identify the % of interpreted code vs the jited code for both the binaries. We are also working on creating a pull request to get a better feedback on the change.

Regards Yash

________________________________________ From: Armin Rigo [armin.rigo@gmail.com] Sent: Wednesday, November 02, 2016 2:18 AM To: Singh, Yashwardhan Cc: pypy-dev@python.org Subject: Re: [pypy-dev] PGO Optimized Binary

Hi,

On 31 October 2016 at 22:28, Singh, Yashwardhan <yashwardhan.singh@intel.com> wrote:

...

We applied compiler assisted optimization technique called PGO or Profile Guided Optimization while building PyPy, and found performance got improved by up to 22.4% on the Grand Unified Python Benchmark (GUPB) from ?hg clone https://hg.python.org/benchmarks?. The below result table shows majority of 51 micros got performance boost with 8 got performance regression.

The kind of performance improvement you are measuring involves only short- or very short-running programs. A few years ago we'd have shrugged it off as irrelevant---"please modify the benchmarks so that they run for at least 10 seconds, more if they are larger"---because the JIT compiler doesn't have a chance to warm up. But we'd also have shrugged off your whole attempt---"PGO optimization cannot change anything to the speed of JIT-produced machine code".

Nowadays we tend to look more seriously at the cold or warming-up performance too, or at least we know that we should look there. There are (stalled) plans of setting up a second benchmark suite for PyPy which focuses on this.

You can get an estimate of whether you're looking at cold or hot code: compare the timings with CPython. Also, you can set the environment variable ``PYPYLOG=jit-summary:-`` and look at the first 2 lines to see how much time was spent warming up the JIT (or attempting to).

Note that we did enable PGO long ago, with modest benefits. We gave up when our JIT compiler became good enough. Maybe now is the time to try again (and also, PGO itself might have improved in the meantime).

...

We?d like to get some input on how to contribute our optimization recipe to the PyPy dev tree, perhaps by creating an item to the PyPy issue tracker?

The best would be to create a pull request so that we can look at your changes more easily.

...

In addition, we would also appreciate any other benchmark or real world use based workload as alternatives to evaluate this.

You can take any Python program that runs either very shortly or not faster than CPython. For a larger example (with Python 2.7):

cd rpython/jit/tl python ../../bin/rpython -O2 --source targettlr # 24 secs pypy ../../bin/rpython -O2 --source targettlr # 39 secs

A bient?t,

Armin. _______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev

------------------------------ Message: 2 Date: Fri, 11 Nov 2016 11:56:24 +0200 From: Matti Picus <matti.picus@gmail.com> To: "pypy-dev@python.org" <pypy-dev@python.org> Subject: [pypy-dev] speed.pypy.org and old versions Message-ID: <b2ea7e6e-9f7b-0d0f-0fa8-7bcdfbf13a97@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed The landing page for speed.pypy.org, and the pull-downs on the other pages, are clogged with old results from previous versions. It seems the comparison page http://speed.pypy.org/comparison cannot even pull up the database info, and times out, making it useless. A quick fix could be to remove tags for much of the historical data, say for things more than 3 years old, keeping a smattering of significant versions? What you all think? Matti ------------------------------ Message: 3 Date: Fri, 11 Nov 2016 12:46:19 +0200 From: Matti Picus <matti.picus@gmail.com> To: "pypy-dev@python.org" <pypy-dev@python.org> Subject: [pypy-dev] release tarballs of the latest PyPy2.7 available on bitbucket Message-ID: <a36ca1db-1f7a-7ee4-0c66-78b3a4b5f67c@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed https://bitbucket.org/pypy/pypy/downloads Please download and try them out, if there are no problems we will try for a Sunday release Matti ------------------------------ Message: 4 Date: Fri, 11 Nov 2016 14:34:30 +0100 From: Carl Friedrich Bolz <cfbolz@gmx.de> To: Matti Picus <matti.picus@gmail.com>, "pypy-dev@python.org" <pypy-dev@python.org> Subject: Re: [pypy-dev] speed.pypy.org and old versions Message-ID: <60f1f4dc-d085-42a6-b7f4-75579d03da55@email.android.com> Content-Type: text/plain; charset="utf-8" Hi Matti, Sounds good to me! Maybe we can keep the tag info around somewhere else (a file in extradoc?) in case we ever need it again. Thanks for doing this! Carl Friedrich On November 11, 2016 10:56:24 AM GMT+01:00, Matti Picus <matti.picus@gmail.com> wrote:

The landing page for speed.pypy.org, and the pull-downs on the other pages, are clogged with old results from previous versions. It seems the comparison page http://speed.pypy.org/comparison cannot even pull up the database info, and times out, making it useless. A quick fix could be to remove tags for much of the historical data, say for things more than 3 years old, keeping a smattering of significant versions? What you all think? Matti _______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev