Mailman 3 September 2016 - Speed

perf 0.7.11 released
by Victor Stinner Sept. 19, 2016

Sept. 19, 2016

Hi, I released perf 0.7.11. News since perf 0.7.3: * Support PyPy * Add units to samples: second, byte, integer. Benchmarks on the memory usage (track memory) are now displayed correctly. * Remove environment variables: add --inherit-environ cmdline option. * Add more metadata: mem_max_rss, python_hash_seed (PYTHONHASHSEED env var). Enhance cpu_conifig: add nohz_full & isolated. Enhance python_version: add Mercurial revision. * Better and more reliable code to calibrate benchmarks, calibration samples are now stored (in warmup samples). * Bugfixes as usual. perf changelog: Version 0.7.11 (2016-09-19) --------------------------- * Fix metadata when NOHZ is not used: when /sys/devices/system/cpu/nohz_full contains ' (null)\n' Version 0.7.10 (2016-09-17) --------------------------- * Fix metadata when there is no isolated CPU * Fix collecting metadata when /sys/devices/system/cpu/nohz_full doesn't exist Version 0.7.9 (2016-09-17) -------------------------- * Add :meth:`Benchmark.get_unit` method * Add :meth:`BenchmarkSuite.get_metadata` method * metadata: add ``nohz_full`` and ``isolated`` to ``cpu_config`` * add ``--affinity`` option to the ``metadata`` command * ``convert``: fix ``--remove-all-metadata``, keep the unit * metadata: fix regex to get the Mercurial revision for ``python_version``, support also locally modified source code (revision ending with "+") Version 0.7.8 (2016-09-10) -------------------------- * Worker child processes are now run in a fresh environment: environment variables are removed, to enhance reproductability. * Add ``--inherit-environ`` command line argument. * metadata: add ``python_cflags``, fix ``python_version`` for PyPy and add also the Mercurial version into ``python_version`` (if available) Version 0.7.7 (2016-09-07) -------------------------- * Reintroduce TextRunner._spawn_worker_suite() as a temporary workaround to fix the pybench benchmark of the performance module. Version 0.7.6 (2016-09-02) -------------------------- Tracking memory usage now works correctly on Linux and Windows. The calibration is now done in a the first worker process. * ``--tracemalloc`` and ``--track-memory`` now use the memory peak as the unique sample for the run. * Rewrite code to track memory usage on Windows. Add ``mem_peak_pagefile_usage`` metadata. The ``win32api`` module is no more needed, the code now uses the ``ctypes`` module. * ``convert``: add ``--remove-all-metadata`` and ``--update-metadata`` commands * Add ``unit`` metadata: ``byte``, ``integer`` or ``second``. * Run samples can now be integer (not only float). * Don't round samples to 1 nanosecond anymore: with a large number of loops (ex: 2^24), rounding reduces the accuracy. * The benchmark calibration is now done by the first worker process Version 0.7.5 (2016-09-01) -------------------------- * Add ``Benchmark.update_metadata()`` method * Warmup samples can now be zero. TextRunner now raises an error if a sample function returns zero for a sample, except of calibration and warmup samples. Version 0.7.4 (2016-08-18) -------------------------- * Support PyPy * metadata: add ``mem_max_rss`` and ``python_hash_seed`` * Add :func:`perf.python_implementation` and :func:`perf.python_has_jit` functions * In workers, calibration samples are now stored as warmup samples. * With a JIT (PyPy), the calibration is now done in each worker. The warmup step can compute more warmup samples if a raw sample is shorter than the minimum time. * Warmups of Run objects are now lists of (loops, raw_sample) rather than lists of samples. This change requires a change in the JSON format. Victor

1 0

performance 0.2.2 released
by Victor Stinner Sept. 19, 2016

Sept. 19, 2016

Hi, I released performance 0.2.2. Compared to performance 0.1: * it fixes the --track-memory option, adds a new "show" command, * enhance "compare" output (display python & performance versions, use sample units: seconds or bytes), * it isolates again environment variables (fix --inherit-environ cmdline option), * bugfixes as usual. Version 0.2.2 (2016-09-19) -------------------------- * Add a new ``show`` command to display a benchmark files * Issue #11: Display Python version in compare. Display also the performance version. * CPython issue #26383; csv output: don't truncate digits for timings shorter than 1 us * compare: Use sample unit of benchmarks, format values in the table output using the unit * compare: Fix the table output if benchmarks only contain a single sample * Remove unused -C/--control_label and -E/--experiment_label options * Update perf dependency to 0.7.11 to get Benchmark.get_unit() and BenchmarkSuite.get_metadata() Version 0.2.1 (2016-09-10) -------------------------- * Add ``--csv`` option to the ``compare`` command * Fix ``compare -O table`` output format * Freeze indirect dependencies in requirements.txt * ``run``: add ``--track-memory`` option to track the memory peak usage * Update perf dependency to 0.7.8 to support memory tracking and the new ``--inherit-environ`` command line option * If ``virtualenv`` command fail, try another command to create the virtual environment: catch ``virtualenv`` error * The first command to upgrade pip to version ``>= 6.0`` now uses the ``pip`` binary rather than ``python -m pip`` to support pip 1.0 which doesn't support ``python -m pip`` CLI. * Update Django (1.10.1), Mercurial (3.9.1) and psutil (4.3.1) * Rename ``--inherit_env`` command line option to ``--inherit-environ`` and fix it Version 0.2 (2016-09-01) ------------------------ * Update Django dependency to 1.10 * Update Chameleon dependency to 2.24 * Add the ``--venv`` command line option * Convert Python startup, Mercurial startup and 2to3 benchmarks to perf scripts (bm_startup.py, bm_hg_startup.py and bm_2to3.py) * Pass the ``--affinity`` option to perf scripts rather than using the ``taskset`` command * Put more installer and optional requirements into ``performance/requirements.txt`` * Cached ``.pyc`` files are no more removed before running a benchmark. Use ``venv recreate`` command to update a virtual environment if required. * The broken ``--track_memory`` option has been removed. It will be added back when it will be fixed. * Add performance version to metadata * Upgrade perf dependency to 0.7.5 to get ``Benchmark.update_metadata()`` Victor

1 0

Re: [Speed] New instance of CodeSpeed at speed.python.org running performance on CPython and PyPy?
by Victor Stinner Sept. 1, 2016

Sept. 1, 2016

2016-09-01 19:53 GMT+02:00 Kevin Modzelewski <kmod(a)dropbox.com>: > Just my two cents -- having a benchmark change underneath the benchmark > runner is quite confusing to debug, because it looks indistinguishable from > a non-reproducible regression that happens in the performance itself. I agree. That's why I proposed to use semantic versionning. I'm not sure that old results must be removed. We should just be explicit about versions. The main issue is when you *compare* two results produced by two different performance versions. I have an item in my TODO list to emit a warning if the exact version (minor version) is different, and display an error if the major version is different. About reproductability: I made another change in the development version, indirect dependencies are now pinned as well: https://github.com/python/performance/blob/master/performance/requirements.… It should help to have a more reproductible benchmark ;-) The last known issue about reproductability is that I dropped the code to remove environment variables. I should fix this in the perf module directly. Interesting link: https://reproducible-builds.org/ > Another thing to consider is that there will be other people using this > benchmark set than just the codespeed setup: there will be long-lived > benchmark results in the form of blogs and academic papers. I think it's > important to get some good wording about having to include the version of > the benchmarks when publishing results, and then it would be good to follow > that advice internally as well. The perf module has a feature for that: it supports storing metadata in JSON files. I modified many benchmarks to store dependency version (like Django verison), but now also store performance version (since performance 0.2). The perf module stores its own version by default ;-) I suggest to store JSON file rather than only the compact text output (which contains much less information). Victor

1 0

Re: [Speed] New instance of CodeSpeed at speed.python.org running performance on CPython and PyPy?
by Kevin Modzelewski Sept. 1, 2016

Sept. 1, 2016

Just my two cents -- having a benchmark change underneath the benchmark runner is quite confusing to debug, because it looks indistinguishable from a non-reproducible regression that happens in the performance itself. My vote would be to wipe the benchmark results when this happens (and if that is too expensive, not upgrade that often). Another thing to consider is that there will be other people using this benchmark set than just the codespeed setup: there will be long-lived benchmark results in the form of blogs and academic papers. I think it's important to get some good wording about having to include the version of the benchmarks when publishing results, and then it would be good to follow that advice internally as well. kmod On Thu, Sep 1, 2016 at 3:58 AM, Victor Stinner <victor.stinner(a)gmail.com> wrote: > Hi, > > Would it be possible to run a new instance of CodeSpeed (the website > behing speed.python.org) which would run the "performance" benchmark > suite rather than the "benchmarks" benchmark suite? And would it be > possible to run it on CPython (2.7 and 3.5 branches) and PyPy (master > branch, maybe also the py3k branch)? > > I found https://github.com/tobami/codespeed/ but I didn't look at it > right now. I guess that some code should be written to convert perf > JSON file to the format expected by CodeSpeed? > > FYI I released performance 0.2 yesterday. JSON files now contain the > version of the benchmark suite ("performance_version: 0.2"). I plan to > use semantic version: increase the major version (ex: upgrade to 0.3, > but later it will be 1.x, 2.x, etc.) when benchmark results are > considered to not be compatible. > > For example, I upgraded Django (from 1.9 to 1.10) and Chameleon (from > 2.22 to 2.24) in performance 0.2. > > The question is how to upgrade the performance to a new major version: > should we drop previous benchmark results? > > Maybe we should put the performance version in the URL, and use > "/latest/" by default. Only /latest/ would get new results, and > /latest/ would restart from an empty set of results when performance > is upgraded? > > Another option, less exciting, is to never upgrade benchmarks. The > benchmarks project *added* new benchmarks when a dependency was > "upgraded". In fact, the old dependency was kept and a new dependency > (full copy of the code in fact ;-)) was added. So it has django, > django_v2, django_v3, etc. The problem is that it still uses Mercurial > 1.2 which was released 7 years ago (2009)... Since it's painful to > upgrade, most dependencies were outdated. > > Do you care of old benchmark results? It's quite easy to regenerate > them (on demand?) if needed, no? Using Mercurial and Git, it's easy to > update to any old revisions to run again a benchmark on an old version > of CPython / PyPy / etc. > > Victor > _______________________________________________ > Speed mailing list > Speed(a)python.org > https://mail.python.org/mailman/listinfo/speed >

1 0

Re: [Speed] New instance of CodeSpeed at speed.python.org running performance on CPython and PyPy?
by Zachary Ware Sept. 1, 2016

Sept. 1, 2016

On Thu, Sep 1, 2016 at 5:58 AM, Victor Stinner <victor.stinner(a)gmail.com> wrote: > Hi, > > Would it be possible to run a new instance of CodeSpeed (the website > behing speed.python.org) which would run the "performance" benchmark > suite rather than the "benchmarks" benchmark suite? And would it be > possible to run it on CPython (2.7 and 3.5 branches) and PyPy (master > branch, maybe also the py3k branch)? Short answer is yes, please :). Slightly longer answer is that that's the plan, but I don't know when I'll have opportunity to work on it. Possibly next week at the sprint, we'll see. > I found https://github.com/tobami/codespeed/ but I didn't look at it > right now. I guess that some code should be written to convert perf > JSON file to the format expected by CodeSpeed? The code that's actually running speed.python.org is at https://github.com/zware/codespeed, speed.python.org branch. I've been meaning to get that moved to https://github.com/python/codespeed, but it hasn't happened yet. Other relevant code is hidden in the buildbot master and on the runner box itself, which is not publicly version controlled (which is bad). We will need either a translation layer between performance and CodeSpeed, or if we can, just change the format that performance outputs to match what CodeSpeed expects. > FYI I released performance 0.2 yesterday. JSON files now contain the > version of the benchmark suite ("performance_version: 0.2"). I plan to > use semantic version: increase the major version (ex: upgrade to 0.3, > but later it will be 1.x, 2.x, etc.) when benchmark results are > considered to not be compatible. > > For example, I upgraded Django (from 1.9 to 1.10) and Chameleon (from > 2.22 to 2.24) in performance 0.2. > > The question is how to upgrade the performance to a new major version: > should we drop previous benchmark results? > > Maybe we should put the performance version in the URL, and use > "/latest/" by default. Only /latest/ would get new results, and > /latest/ would restart from an empty set of results when performance > is upgraded? I have only enough experience with Django and CodeSpeed to have gotten speed.python.org to the state that it currently is, so I really don't know how (un)limited the possibilities are. One simple method would be to combine the benchmark name with the performance version, and periodically clear out old benchmark results. > Another option, less exciting, is to never upgrade benchmarks. The > benchmarks project *added* new benchmarks when a dependency was > "upgraded". In fact, the old dependency was kept and a new dependency > (full copy of the code in fact ;-)) was added. So it has django, > django_v2, django_v3, etc. The problem is that it still uses Mercurial > 1.2 which was released 7 years ago (2009)... Since it's painful to > upgrade, most dependencies were outdated. I agree that we should have the ability to easily update benchmarks and actually do so sometimes. > Do you care of old benchmark results? It's quite easy to regenerate > them (on demand?) if needed, no? Using Mercurial and Git, it's easy to > update to any old revisions to run again a benchmark on an old version > of CPython / PyPy / etc. I suggest that upon updates to the benchmark suite/runner/etc., we should clear out old results and rerun the benchmarks on a selection of released versions of each interpreter. We should also have some way to trigger a run of the benchmarks on a particular revision of an interpreter. -- Zach

1 0

Re: [Speed] New instance of CodeSpeed at speed.python.org running performance on CPython and PyPy?
by Brett Cannon Sept. 1, 2016

Sept. 1, 2016

On Thu, 1 Sep 2016 at 03:58 Victor Stinner <victor.stinner(a)gmail.com> wrote: > Hi, > > Would it be possible to run a new instance of CodeSpeed (the website > behing speed.python.org) which would run the "performance" benchmark > suite rather than the "benchmarks" benchmark suite? And would it be > possible to run it on CPython (2.7 and 3.5 branches) and PyPy (master > branch, maybe also the py3k branch)? > I believe Zach has the repo containing the code. He also said it's all rather hacked up at the moment. Maybe something to discuss next week at the sprint as I think you're both going to be there. > > I found https://github.com/tobami/codespeed/ but I didn't look at it > right now. I guess that some code should be written to convert perf > JSON file to the format expected by CodeSpeed? > > FYI I released performance 0.2 yesterday. JSON files now contain the > version of the benchmark suite ("performance_version: 0.2"). I plan to > use semantic version: increase the major version (ex: upgrade to 0.3, > but later it will be 1.x, 2.x, etc.) when benchmark results are > considered to not be compatible. > SGTM. > > For example, I upgraded Django (from 1.9 to 1.10) and Chameleon (from > 2.22 to 2.24) in performance 0.2. > > The question is how to upgrade the performance to a new major version: > should we drop previous benchmark results? > They don't really compare anymore, so they should at least not be compared to benchmark results from a newer benchmark. > > Maybe we should put the performance version in the URL, and use > "/latest/" by default. Only /latest/ would get new results, and > /latest/ would restart from an empty set of results when performance > is upgraded? > SGTM > > Another option, less exciting, is to never upgrade benchmarks. The > benchmarks project *added* new benchmarks when a dependency was > "upgraded". In fact, the old dependency was kept and a new dependency > (full copy of the code in fact ;-)) was added. So it has django, > django_v2, django_v3, etc. The problem is that it still uses Mercurial > 1.2 which was released 7 years ago (2009)... Since it's painful to > upgrade, most dependencies were outdated. > Based on my experience with the benchmark suite I don't like this option either; it just gathers cruft. As Maciej and the PyPy folks have pointed out, benchmarks should try to represent modern code and old benchmarks won't necessarily do that. > > Do you care of old benchmark results? It's quite easy to regenerate > them (on demand?) if needed, no? Using Mercurial and Git, it's easy to > update to any old revisions to run again a benchmark on an old version > of CPython / PyPy / etc. > I personally don't, but that's because care about either current performance in comparison to others or very short timescales to see when a regression occurred (hence a switchover has a very small chance of impacting that investigation), not long timescale results for historical purposes.

1 0

New instance of CodeSpeed at speed.python.org running performance on CPython and PyPy?
by Victor Stinner Sept. 1, 2016

Sept. 1, 2016

Hi, Would it be possible to run a new instance of CodeSpeed (the website behing speed.python.org) which would run the "performance" benchmark suite rather than the "benchmarks" benchmark suite? And would it be possible to run it on CPython (2.7 and 3.5 branches) and PyPy (master branch, maybe also the py3k branch)? I found https://github.com/tobami/codespeed/ but I didn't look at it right now. I guess that some code should be written to convert perf JSON file to the format expected by CodeSpeed? FYI I released performance 0.2 yesterday. JSON files now contain the version of the benchmark suite ("performance_version: 0.2"). I plan to use semantic version: increase the major version (ex: upgrade to 0.3, but later it will be 1.x, 2.x, etc.) when benchmark results are considered to not be compatible. For example, I upgraded Django (from 1.9 to 1.10) and Chameleon (from 2.22 to 2.24) in performance 0.2. The question is how to upgrade the performance to a new major version: should we drop previous benchmark results? Maybe we should put the performance version in the URL, and use "/latest/" by default. Only /latest/ would get new results, and /latest/ would restart from an empty set of results when performance is upgraded? Another option, less exciting, is to never upgrade benchmarks. The benchmarks project *added* new benchmarks when a dependency was "upgraded". In fact, the old dependency was kept and a new dependency (full copy of the code in fact ;-)) was added. So it has django, django_v2, django_v3, etc. The problem is that it still uses Mercurial 1.2 which was released 7 years ago (2009)... Since it's painful to upgrade, most dependencies were outdated. Do you care of old benchmark results? It's quite easy to regenerate them (on demand?) if needed, no? Using Mercurial and Git, it's easy to update to any old revisions to run again a benchmark on an old version of CPython / PyPy / etc. Victor

1 0