Mailman 3 August 2016 - Speed

Re: [Speed] New benchmark suite for Python
by Nick Coghlan Aug. 21, 2016

Aug. 21, 2016

On 20 August 2016 at 02:50, Maciej Fijalkowski <fijall(a)gmail.com> wrote: > Very likely just pyc import time As one of the import system maintainers, that's a number I consider quite interesting and worth benchmarking :) It's also one of the key numbers for Linux distro Python usage, since it impacts how responsive the system shell feels to developers and administrators - an end user can't readily tell the difference between "this shell is slow" and "this particular command I am running is using a language interpreter with a long startup time", but an interpreter benchmark suite can. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Re: [Speed] New benchmark suite for Python
by Maciej Fijalkowski Aug. 19, 2016

Aug. 19, 2016

Very likely just pyc import time On Fri, Aug 19, 2016 at 6:48 PM, Alex Gaynor <alex.gaynor(a)gmail.com> wrote: > It's probably an ok benchmark of warmup. > > Alex > > On Fri, Aug 19, 2016 at 12:47 PM, Maciej Fijalkowski <fijall(a)gmail.com> > wrote: >> >> On Fri, Aug 19, 2016 at 1:20 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: >> > On 19 August 2016 at 01:55, Victor Stinner <victor.stinner(a)gmail.com> >> > wrote: >> >> 2016-08-18 8:48 GMT+02:00 Armin Rigo <arigo(a)tunes.org>: >> >>> Indeed, bzr cannot be installed on PyPy because it uses Cython in a >> >>> strange way: it declares and directly pokes inside PyListObjects from >> >>> a .pyx file. But note that bzr (seems to) have systematically a pure >> >>> Python version of all its .pyx files. (...) >> >> >> >> bazar is only used for a "startup" benchmark. I don't think that such >> >> benchmark is very interesting... I would prefer to see a benchmark on >> >> a less dummy operation on the repository than displaying the help... >> > >> > Simple commands like displaying help messages are where interpreter >> > startup time dominates the end user experience for applications >> > written in Python, though. For example, improvements to import system >> > performance tend to mostly show up there - for longer running >> > benchmarks, changes in startup time tend to get swamped by the actual >> > runtime speed, while the baseline "python -c 'pass'" mainly varies >> > based on how many modules we're implicitly importing at startup rather >> > than how well the import system is performing . >> > >> > Cheers, >> > Nick. >> >> I would still argue that displaying help is not a very good benchmark :-) >> _______________________________________________ >> Speed mailing list >> Speed(a)python.org >> https://mail.python.org/mailman/listinfo/speed > > > > > -- > "I disapprove of what you say, but I will defend to the death your right to > say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > GPG Key fingerprint: D1B3 ADC0 E023 8CA6 >

1 0

Re: [Speed] New benchmark suite for Python
by Alex Gaynor Aug. 19, 2016

Aug. 19, 2016

It's probably an ok benchmark of warmup. Alex On Fri, Aug 19, 2016 at 12:47 PM, Maciej Fijalkowski <fijall(a)gmail.com> wrote: > On Fri, Aug 19, 2016 at 1:20 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > > On 19 August 2016 at 01:55, Victor Stinner <victor.stinner(a)gmail.com> > wrote: > >> 2016-08-18 8:48 GMT+02:00 Armin Rigo <arigo(a)tunes.org>: > >>> Indeed, bzr cannot be installed on PyPy because it uses Cython in a > >>> strange way: it declares and directly pokes inside PyListObjects from > >>> a .pyx file. But note that bzr (seems to) have systematically a pure > >>> Python version of all its .pyx files. (...) > >> > >> bazar is only used for a "startup" benchmark. I don't think that such > >> benchmark is very interesting... I would prefer to see a benchmark on > >> a less dummy operation on the repository than displaying the help... > > > > Simple commands like displaying help messages are where interpreter > > startup time dominates the end user experience for applications > > written in Python, though. For example, improvements to import system > > performance tend to mostly show up there - for longer running > > benchmarks, changes in startup time tend to get swamped by the actual > > runtime speed, while the baseline "python -c 'pass'" mainly varies > > based on how many modules we're implicitly importing at startup rather > > than how well the import system is performing . > > > > Cheers, > > Nick. > > I would still argue that displaying help is not a very good benchmark :-) > _______________________________________________ > Speed mailing list > Speed(a)python.org > https://mail.python.org/mailman/listinfo/speed > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: D1B3 ADC0 E023 8CA6

1 0

Re: [Speed] New benchmark suite for Python
by Maciej Fijalkowski Aug. 19, 2016

Aug. 19, 2016

On Fri, Aug 19, 2016 at 1:20 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > On 19 August 2016 at 01:55, Victor Stinner <victor.stinner(a)gmail.com> wrote: >> 2016-08-18 8:48 GMT+02:00 Armin Rigo <arigo(a)tunes.org>: >>> Indeed, bzr cannot be installed on PyPy because it uses Cython in a >>> strange way: it declares and directly pokes inside PyListObjects from >>> a .pyx file. But note that bzr (seems to) have systematically a pure >>> Python version of all its .pyx files. (...) >> >> bazar is only used for a "startup" benchmark. I don't think that such >> benchmark is very interesting... I would prefer to see a benchmark on >> a less dummy operation on the repository than displaying the help... > > Simple commands like displaying help messages are where interpreter > startup time dominates the end user experience for applications > written in Python, though. For example, improvements to import system > performance tend to mostly show up there - for longer running > benchmarks, changes in startup time tend to get swamped by the actual > runtime speed, while the baseline "python -c 'pass'" mainly varies > based on how many modules we're implicitly importing at startup rather > than how well the import system is performing . > > Cheers, > Nick. I would still argue that displaying help is not a very good benchmark :-)

1 0

Re: [Speed] New benchmark suite for Python
by Nick Coghlan Aug. 19, 2016

Aug. 19, 2016

On 19 August 2016 at 01:55, Victor Stinner <victor.stinner(a)gmail.com> wrote: > 2016-08-18 8:48 GMT+02:00 Armin Rigo <arigo(a)tunes.org>: >> Indeed, bzr cannot be installed on PyPy because it uses Cython in a >> strange way: it declares and directly pokes inside PyListObjects from >> a .pyx file. But note that bzr (seems to) have systematically a pure >> Python version of all its .pyx files. (...) > > bazar is only used for a "startup" benchmark. I don't think that such > benchmark is very interesting... I would prefer to see a benchmark on > a less dummy operation on the repository than displaying the help... Simple commands like displaying help messages are where interpreter startup time dominates the end user experience for applications written in Python, though. For example, improvements to import system performance tend to mostly show up there - for longer running benchmarks, changes in startup time tend to get swamped by the actual runtime speed, while the baseline "python -c 'pass'" mainly varies based on how many modules we're implicitly importing at startup rather than how well the import system is performing . Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Re: [Speed] New benchmark suite for Python
by Brett Cannon Aug. 18, 2016

Aug. 18, 2016

On Wed, 17 Aug 2016 at 17:37 Victor Stinner <victor.stinner(a)gmail.com> wrote: > Hi, > > After a few months of work, I created a "new" benchmark suite for Python: > > https://github.com/python/benchmarks > [SNIP] > Yay! > > Known regressions: > > * Memory tracking is broken > * run_compare command is currently broken: use run (store result into > a file) + compare manually > * Some benchmarks have been removed: rietveld, spitfire (not on PyPI), > pystone, gcbench, tuple_gc_hell > * I only tested Linux, I expect issues on Windows. (I didn't try my > perf module on Windows yet.) > > I already allowed all Python core developers to push to the GitHub > project. We can create a new "benchmarks" (or "Performance" maybe?) > team if we want to allow more contributors who are not core > developers. > > PyPy, Pyston, Pyjion, Numba, etc. : Hey! it's now time to start to > take a look at my project and test it ;-) Tell me what is broken, what > is missing, and I will try to help you to move your project to this > new benchmark suite! > > As requested (suggested?) by Brett Cannon, the Git repository has no > history, it only contains 1 commit! I'm really sorry of losing all > the history and all authors, but it allows to start with a much > smaller repository: around 2 MB. The current benchmark repository is > more around 200 MB! > If people care then they can look at hg.python.org/benchmarks > > TODO: > > * continue to upgrade libraries in requirements.txt. I failed to > upgrade Django to 1.10, it complains about a missing template engine > config setting. > * convert more code to the perf module, like "startup" tests > * run benchmarks and analyze results ;-) > * write more documentation explaining how to run reliable benchmarks > * ... > I'll try to get around and run the benchmarks on Windows to see if any issues come up.

1 0

Re: [Speed] New benchmark suite for Python
by Victor Stinner Aug. 18, 2016

Aug. 18, 2016

2016-08-18 3:17 GMT+02:00 Victor Stinner <victor.stinner(a)gmail.com>: > I made a quick test on PyPy: (...) I ran a longer test last night. Some benchmarks were slower on PyPy. In fact, the benchmark doesn't give enough time to PyPy to warm up its JIT. I just released perf 0.7.4 which has a better calibration code for Python using a JIT (PyPy), better default configuration values: Default (with a JIT, ex: PyPy): 6 processes, 10 samples per process (total: 60 samples), and 10 warmups. vs Default (no JIT, ex: CPython): 20 processes, 3 samples per process (total: 60 samples), and 1 warmup. perf 0.7.4 also has new helper functions: python_implementation() and python_has_jit(). I started to patch benchmarks to increase the number of warmups (even more) of some benchmarks: * go: 50 warmups * hexiom2: 15 warmups * tornado http: 30 warmups If perf detects a JIT (PyPy), the warmup step now computes more samples dynamically if it detects that a raw sample is smaller than the minimum time (100 ms). All these changes were written to help PyPy to warm up its JIT. I'm not sure that it's fully correct, it may make benchmarks less reliable since the number of warmup samples is no more constant. Maybe the code should be enhanced even more to at least use the same parameters in all worker processes. Victor

1 0

Re: [Speed] New benchmark suite for Python
by Victor Stinner Aug. 18, 2016

Aug. 18, 2016

Hi Maciej, 2016-08-18 9:17 GMT+02:00 Maciej Fijalkowski <fijall(a)gmail.com>: > Did you look into integrating the pypy benchmarks that we added over the years? Not yet, but yes, I plan to collect benchmarks from PyPy, Pyston, Pyjion, etc. Later we can discuss if some benchmarks should be disabled in the default set of benchmarks. Victor

1 0

Re: [Speed] New benchmark suite for Python
by Victor Stinner Aug. 18, 2016

Aug. 18, 2016

2016-08-18 8:48 GMT+02:00 Armin Rigo <arigo(a)tunes.org>: > Indeed, bzr cannot be installed on PyPy because it uses Cython in a > strange way: it declares and directly pokes inside PyListObjects from > a .pyx file. But note that bzr (seems to) have systematically a pure > Python version of all its .pyx files. (...) bazar is only used for a "startup" benchmark. I don't think that such benchmark is very interesting... I would prefer to see a benchmark on a less dummy operation on the repository than displaying the help... Victor

1 0

Re: [Speed] New benchmark suite for Python
by Victor Stinner Aug. 18, 2016

Aug. 18, 2016

2016-08-18 6:47 GMT+02:00 Zachary Ware <zachary.ware+pydev(a)gmail.com>: > Also, if you're interested in having your interpreter benchmarked on > speed.python.org, contact me with clear instructions (preferably in > the form of a shell or Python script) on how to build, test, install, > and invoke your interpreter from a fresh Ubuntu 16.04 installation. Cool. I would prefer to wait a few days until people have the opportunity of test the new benchmark suite on their computer and one different platforms. I'm not sure that it's stable yet. For example, I just modified the telco benchmark to use I/O in memory, rather than using the filesystem (disk). Victor

1 0