Mailman 3 August 2016 - Speed

Re: [Speed] New benchmark suite for Python
by Stefan Behnel Aug. 18, 2016

Aug. 18, 2016

Zachary Ware schrieb am 18.08.2016 um 06:47: > On Wed, Aug 17, 2016 at 7:37 PM, Victor Stinner wrote: >> PyPy, Pyston, Pyjion, Numba, etc. : Hey! it's now time to start to >> take a look at my project and test it ;-) Tell me what is broken, what >> is missing, and I will try to help you to move your project to this >> new benchmark suite! > > Also, if you're interested in having your interpreter benchmarked on > speed.python.org, contact me with clear instructions (preferably in > the form of a shell or Python script) on how to build, test, install, > and invoke your interpreter from a fresh Ubuntu 16.04 installation. > As an example, here's an untested version for CPython 3.x: > > #!/bin/sh > # set up dependencies > sudo apt-get build-dep -y python3 > sudo apt-get install -y --no-install-recommends mercurial > # get the code > hg clone https://hg.python.org/cpython > cd cpython > # build > ./configure --prefix=/opt/python/default > make -j12 > # test > make buildbottest TESTOPTS=-j12 > # install > make install > # invoke > /opt/python/default/bin/python3 > > > I don't know when I'll have a chance to work on it, but I'd like to > get as many projects as possible benchmarked on speed.python.org. Is there a repository somewhere with existing runner scripts that I could look at and send a pull request to? I saw the python/speed.python.org project on github, but that seems rather dead. > Victor: Thanks for getting the new repository set up and for all your > work on the new runner! I'm looking forward to trying it out. +1 Stefan

2 1

Re: [Speed] New benchmark suite for Python
by Maciej Fijalkowski Aug. 18, 2016

Aug. 18, 2016

Hey Victor Did you look into integrating the pypy benchmarks that we added over the years? On Thu, Aug 18, 2016 at 9:37 AM, Victor Stinner <victor.stinner(a)gmail.com> wrote: > Hi, > > After a few months of work, I created a "new" benchmark suite for Python: > > https://github.com/python/benchmarks > > It's based on: > https://hg.python.org/sandbox/benchmarks_perf > which is my fork the CPython benchmark suite: > https://hg.python.org/benchmarks > which is based on Unladen Swallow's benchmark suite (if I understood correctly). > > Major differences: > > * Use the perf module to run benchmarks in multiple processes and > store results as JSON > * Create virtual environments using requirements.txt to download > dependencies from PyPI (rather than using old copies of libraries) > * Many libraries have been upgraded: see requirements.txt > > The project works on Python 2 and Python 3 (I tested 2.7 and 3.6). > > Known regressions: > > * Memory tracking is broken > * run_compare command is currently broken: use run (store result into > a file) + compare manually > * Some benchmarks have been removed: rietveld, spitfire (not on PyPI), > pystone, gcbench, tuple_gc_hell > * I only tested Linux, I expect issues on Windows. (I didn't try my > perf module on Windows yet.) > > I already allowed all Python core developers to push to the GitHub > project. We can create a new "benchmarks" (or "Performance" maybe?) > team if we want to allow more contributors who are not core > developers. > > PyPy, Pyston, Pyjion, Numba, etc. : Hey! it's now time to start to > take a look at my project and test it ;-) Tell me what is broken, what > is missing, and I will try to help you to move your project to this > new benchmark suite! > > As requested (suggested?) by Brett Canon, the Git repository has no > history, it only contains 1 commit! I'm really sorry of loosing all > the history and all authors, but it allows to start with a much > smaller repository: around 2 MB. The current benchmark repository is > more around 200 MB! > > TODO: > > * continue to upgrade libraries in requirements.txt. I failed to > upgrade Django to 1.10, it complains about a missing template engine > config setting. > * convert more code to the perf module, like "startup" tests > * run benchmarks and analyze results ;-) > * write more documentation explaining how to run reliable benchmarks > * ... > > Victor > _______________________________________________ > Speed mailing list > Speed(a)python.org > https://mail.python.org/mailman/listinfo/speed

1 0

Re: [Speed] New benchmark suite for Python
by Armin Rigo Aug. 18, 2016

Aug. 18, 2016

Hi Victor, On 18 August 2016 at 03:17, Victor Stinner <victor.stinner(a)gmail.com> wrote: > I made a quick test on PyPy: the creation of the virtual environment > fails when trying to compile a Cython extension to install bzr. Indeed, bzr cannot be installed on PyPy because it uses Cython in a strange way: it declares and directly pokes inside PyListObjects from a .pyx file. But note that bzr (seems to) have systematically a pure Python version of all its .pyx files. The fix might be as simple as changing setup.py to check for PyPy, and in this case do nothing in add_pyrex_extension(). According to https://answers.launchpad.net/bzr/+faq/703 it should work without any Cython code too. A bientôt, Armin.

1 0

Re: [Speed] New benchmark suite for Python
by Zachary Ware Aug. 18, 2016

Aug. 18, 2016

On Wed, Aug 17, 2016 at 7:37 PM, Victor Stinner <victor.stinner(a)gmail.com> wrote: > PyPy, Pyston, Pyjion, Numba, etc. : Hey! it's now time to start to > take a look at my project and test it ;-) Tell me what is broken, what > is missing, and I will try to help you to move your project to this > new benchmark suite! Also, if you're interested in having your interpreter benchmarked on speed.python.org, contact me with clear instructions (preferably in the form of a shell or Python script) on how to build, test, install, and invoke your interpreter from a fresh Ubuntu 16.04 installation. As an example, here's an untested version for CPython 3.x: #!/bin/sh # set up dependencies sudo apt-get build-dep -y python3 sudo apt-get install -y --no-install-recommends mercurial # get the code hg clone https://hg.python.org/cpython cd cpython # build ./configure --prefix=/opt/python/default make -j12 # test make buildbottest TESTOPTS=-j12 # install make install # invoke /opt/python/default/bin/python3 I don't know when I'll have a chance to work on it, but I'd like to get as many projects as possible benchmarked on speed.python.org. Victor: Thanks for getting the new repository set up and for all your work on the new runner! I'm looking forward to trying it out. -- Zach

1 0

Re: [Speed] New benchmark suite for Python
by Victor Stinner Aug. 18, 2016

Aug. 18, 2016

2016-08-18 2:37 GMT+02:00 Victor Stinner <victor.stinner(a)gmail.com>: > PyPy, Pyston, Pyjion, Numba, etc. : Hey! it's now time to start to > take a look at my project and test it ;-) Tell me what is broken, what > is missing, and I will try to help you to move your project to this > new benchmark suite! I made a quick test on PyPy: the creation of the virtual environment fails when trying to compile a Cython extension to install bzr. Since bzr is not tested by the PyPy benchmark suite, I just removed bzr_startup benchmark to make my Python benchmark suite compatible with PyPy. > As requested (suggested?) by Brett Canon, the Git repository has no > history, ... Reference: https://mail.python.org/pipermail/speed/2016-July/000401.html Brett: "I say just start a new repo from scratch. (...)" Victor

1 0

New benchmark suite for Python
by Victor Stinner Aug. 18, 2016

Aug. 18, 2016

Hi, After a few months of work, I created a "new" benchmark suite for Python: https://github.com/python/benchmarks It's based on: https://hg.python.org/sandbox/benchmarks_perf which is my fork the CPython benchmark suite: https://hg.python.org/benchmarks which is based on Unladen Swallow's benchmark suite (if I understood correctly). Major differences: * Use the perf module to run benchmarks in multiple processes and store results as JSON * Create virtual environments using requirements.txt to download dependencies from PyPI (rather than using old copies of libraries) * Many libraries have been upgraded: see requirements.txt The project works on Python 2 and Python 3 (I tested 2.7 and 3.6). Known regressions: * Memory tracking is broken * run_compare command is currently broken: use run (store result into a file) + compare manually * Some benchmarks have been removed: rietveld, spitfire (not on PyPI), pystone, gcbench, tuple_gc_hell * I only tested Linux, I expect issues on Windows. (I didn't try my perf module on Windows yet.) I already allowed all Python core developers to push to the GitHub project. We can create a new "benchmarks" (or "Performance" maybe?) team if we want to allow more contributors who are not core developers. PyPy, Pyston, Pyjion, Numba, etc. : Hey! it's now time to start to take a look at my project and test it ;-) Tell me what is broken, what is missing, and I will try to help you to move your project to this new benchmark suite! As requested (suggested?) by Brett Canon, the Git repository has no history, it only contains 1 commit! I'm really sorry of loosing all the history and all authors, but it allows to start with a much smaller repository: around 2 MB. The current benchmark repository is more around 200 MB! TODO: * continue to upgrade libraries in requirements.txt. I failed to upgrade Django to 1.10, it complains about a missing template engine config setting. * convert more code to the perf module, like "startup" tests * run benchmarks and analyze results ;-) * write more documentation explaining how to run reliable benchmarks * ... Victor

1 0

Re: [Speed] Tracking memory usage
by Victor Stinner Aug. 17, 2016

Aug. 17, 2016

Ok, I just added the mem_max_rss metadata in the development version of the perf module, just after the perf 0.7.3 release. Victor 2016-08-08 3:38 GMT+02:00 Victor Stinner <victor.stinner(a)gmail.com>: > 2016-07-30 19:48 GMT+02:00 Armin Rigo <arigo(a)tunes.org>: >> Hi Victor, >> >> Fwiw, there is some per-OS (and even apparently >> per-Linux-distribution) solution mentioned here: >> http://stackoverflow.com/questions/774556/peak-memory-usage-of-a-linux-unix… >> >> For me on Arch Linux, "/usr/bin/time -v CMD" returns a reasonable >> value in "Maximum resident set size (kbytes)". I guess that on OSes >> where this works, it gives a zero-overhead, exact answer. > > Oh, I guess that it uses the ru_maxrss field of > getrsage(RUSAGE_CHILDREN). It's also possible to get the maximum RSS > of the current process in pure Python: > >>>> resource.getrusage(resource.RUSAGE_SELF).ru_maxrss > 98700 > > It looks like Linux kernel 2.6.32 or newer is required. Hopefully, > this kernel version is now quite old (December 3rd 2009). > > But I guess that RSS is more coarse than getting the sum of the > private memory from /proc/pid/smaps (Linux 2.6.16 or newer). Sadly, it > looks like the kernel only provides the maximum for RSS memory (not > for private memory). > > Victor

1 0

perf 0.7.3 released
by Victor Stinner Aug. 17, 2016

Aug. 17, 2016

The Python perf module is a toolkit to write, run, analyze and modify benchmarks: http://perf.readthedocs.io/en/latest/ Version 0.7.3 (2016-08-17): * add a new ``slowest`` command * convert: add ``--extract-metadata=NAME`` * add ``--tracemalloc`` option: use the ``tracemalloc`` module to track Python memory allocation and get the peak of memory usage in metadata (``tracemalloc_peak``) * add ``--track-memory`` option: run a thread reading the memory usage every millisecond and store the peak as ``mem_peak`` metadata * ``compare_to``: add ``--group-by-speed`` (``-G``) and ``--min-speed`` options * metadata: add ``runnable_threads`` * Fix issues on ppc64le Power8 Victor

1 0

Re: [Speed] Tracking memory usage
by Victor Stinner Aug. 8, 2016

Aug. 8, 2016

2016-07-30 19:48 GMT+02:00 Armin Rigo <arigo(a)tunes.org>: > Hi Victor, > > Fwiw, there is some per-OS (and even apparently > per-Linux-distribution) solution mentioned here: > http://stackoverflow.com/questions/774556/peak-memory-usage-of-a-linux-unix… > > For me on Arch Linux, "/usr/bin/time -v CMD" returns a reasonable > value in "Maximum resident set size (kbytes)". I guess that on OSes > where this works, it gives a zero-overhead, exact answer. Oh, I guess that it uses the ru_maxrss field of getrsage(RUSAGE_CHILDREN). It's also possible to get the maximum RSS of the current process in pure Python: >>> resource.getrusage(resource.RUSAGE_SELF).ru_maxrss 98700 It looks like Linux kernel 2.6.32 or newer is required. Hopefully, this kernel version is now quite old (December 3rd 2009). But I guess that RSS is more coarse than getting the sum of the private memory from /proc/pid/smaps (Linux 2.6.16 or newer). Sadly, it looks like the kernel only provides the maximum for RSS memory (not for private memory). Victor

1 0