Speed July 2016

speed@python.org

6 participants
24 discussions

by Armin Rigo

Hi Victor, Fwiw, there is some per-OS (and even apparently per-Linux-distribution) solution mentioned here: http://stackoverflow.com/questions/774556/peak-memory-usage-of-a-linux-unix… For me on Arch Linux, "/usr/bin/time -v CMD" returns a reasonable value in "Maximum resident set size (kbytes)". I guess that on OSes where this works, it gives a zero-overhead, exact answer. A bientôt, Armin.

3 years, 10 months

Re: [Speed] CPython Benchmark Suite usinf my perf module, GitHub, etc.

by Zachary Ware

On Fri, Jul 29, 2016 at 4:42 PM, Victor Stinner <victor.stinner(a)gmail.com> wrote: > 2016-07-29 20:51 GMT+02:00 Zachary Ware <zachary.ware+pydev(a)gmail.com>: >> I think rather than using virtual environments which aren't truly >> supported by <3.3 anyway, ... > > What do you mean? I'm building and destroying dozens of venv everyday > at work using tox on Python 2.7. The virtualenv command works well, > no? Do you have issues with it? Not in particular, just that 3.3+ have official support for venvs whereas virtualenv is a bit of a hack by necessity. However, the second point is the real reason I'd rather avoid venvs for this: to make sure that the interpreters actually use the same exact code, so that there can't be any setup.py shenanigans that do things differently between versions/implementations. -- Zach

3 years, 10 months

Re: [Speed] CPython Benchmark Suite usinf my perf module, GitHub, etc.

by Victor Stinner

2016-07-29 20:51 GMT+02:00 Zachary Ware <zachary.ware+pydev(a)gmail.com>: > I think rather than using virtual environments which aren't truly > supported by <3.3 anyway, ... What do you mean? I'm building and destroying dozens of venv everyday at work using tox on Python 2.7. The virtualenv command works well, no? Do you have issues with it? Victor

3 years, 10 months

Re: [Speed] CPython Benchmark Suite usinf my perf module, GitHub, etc.

by Zachary Ware

On Fri, Jul 29, 2016 at 12:03 PM, Brett Cannon <brett(a)python.org> wrote: > On Thu, 28 Jul 2016 at 10:25 Victor Stinner <victor.stinner(a)gmail.com> > wrote: >> * Should we remove vendor copies of libraries and work with virtual >> environments? Not all libraries are available on PyPI :-/ See the >> requirements.txt file and TODO. > > If they are not on PyPI then we should just drop the benchmark. And I say we > do use virtual environments to keep the repo size down. I think rather than using virtual environments which aren't truly supported by <3.3 anyway, we should instead make use of pip's --target, --root, and/or --prefix flags (whatever combination it takes, I haven't looked into it deeply) to install the packages into a particular dir which is then added to each benchmarked interpreter's PYTHONPATH. This way, we're sure that each interpreter is running exactly the same code. Either way, I'm for not vendoring libraries. If the library disappears from PyPI, it's probably not an important workload anymore anyway. >> Reminder: My final goal is to merge again all benchmarks suites >> (CPython, PyPy, Pyston, Pyjion, ...) into one unique project! > > I hope this happens! Me too! I'd also like to get the benchmark runner for speed.python.org set up to build and benchmark as many interpreters as possible. -- Zach

3 years, 10 months

Re: [Speed] CPython Benchmark Suite usinf my perf module, GitHub, etc.

by Victor Stinner

2016-07-29 19:03 GMT+02:00 Brett Cannon <brett(a)python.org>: >> * Should it support --track_memory? it doesn't support it correctly >> right now, it's less precise than before (memory is tracked directly >> in worker processes, no more by the main process) => discuss this >> point in my previous email > > I don't have an opinion as I have never gotten to use the old feature. As I wrote in my other email, I implemented this feature in perf, so the benchmark suite will get it for free. The implementation is not complete, but it's working ;-) >> * Should we remove vendor copies of libraries and work with virtual >> environments? Not all libraries are available on PyPI :-/ See the >> requirements.txt file and TODO. > > If they are not on PyPI then we should just drop the benchmark. And I say we > do use virtual environments to keep the repo size down. Right. We can start with "a subset" of benchmarks and enlarge the test suite later, even reimport old benchmark with their dependency not on PyPI on case by case. Right now, I only wrote requirements.txt. I didn't touch the Python which still "hardcodes" PYTHONPATH to get the local copy of dependencies. It works if you run manually benchmarks in a virtual environment. I will write some glue to automate things and make the code "just work" bfore starting a new thing on GitHub. Victor

3 years, 10 months

Re: [Speed] CPython Benchmark Suite usinf my perf module, GitHub, etc.

by Brett Cannon

On Thu, 28 Jul 2016 at 10:25 Victor Stinner <victor.stinner(a)gmail.com> wrote: > Hi, > > I updated all benchmarks of the CPython Benchmark Suite to use my perf > module. So you get timings of all individual run all *all* benchmarks > and store them in JSON to analyze them in detail. Each benchmark has a > full CLI, for example it gets a new --output option to store result as > JSON directly. But it also gets nice funtions like --hist for > histogram, --stats for statistics, etc. > > The two remaining questions are: > > * Should it support --track_memory? it doesn't support it correctly > right now, it's less precise than before (memory is tracked directly > in worker processes, no more by the main process) => discuss this > point in my previous email > I don't have an opinion as I have never gotten to use the old feature. > > * Should we remove vendor copies of libraries and work with virtual > environments? Not all libraries are available on PyPI :-/ See the > requirements.txt file and TODO. > If they are not on PyPI then we should just drop the benchmark. And I say we do use virtual environments to keep the repo size down. > > My repository: > https://hg.python.org/sandbox/benchmarks_perf > > I would like to push my work as a single giant commit. > > Brett also proposed me to move the benchmarks repository to GitHub > (and so convert it to Git). I don't know if it's appropriate to do all > these things at once? What do you think? > I say just start a new repo from scratch. There isn't a ton of magical history in the hg repo that I think we need to have carried around in the git repo. Plus if we stop shipping project source with the repo then it will be immensely smaller if we start from scratch. > > Reminder: My final goal is to merge again all benchmarks suites > (CPython, PyPy, Pyston, Pyjion, ...) into one unique project! I hope this happens!

3 years, 10 months

Re: [Speed] Tracking memory usage

by Victor Stinner

I modified my perf module to add two new options: --tracemalloc and --track-memory. --tracemalloc enables tracemalloc and gets the peak of the traced Python memory allocations: peak is computed per process. --track-memory is similar but reads PeakPagefileUsage of GetProcessMemoryInfo() on Windows or private data from /proc/self/smap on Linux. The read is done every millisecond (1 ms) in a thread, in the worker process. It's not perfect, but it should be "as good" as the "old" CPython benchmark suite. And it makes the benchmark suite simpler because tracking memory usage is now done automatically by the perf module. Victor

3 years, 10 months

CPython Benchmark Suite usinf my perf module, GitHub, etc.

by Victor Stinner

Hi, I updated all benchmarks of the CPython Benchmark Suite to use my perf module. So you get timings of all individual run all *all* benchmarks and store them in JSON to analyze them in detail. Each benchmark has a full CLI, for example it gets a new --output option to store result as JSON directly. But it also gets nice funtions like --hist for histogram, --stats for statistics, etc. The two remaining questions are: * Should it support --track_memory? it doesn't support it correctly right now, it's less precise than before (memory is tracked directly in worker processes, no more by the main process) => discuss this point in my previous email * Should we remove vendor copies of libraries and work with virtual environments? Not all libraries are available on PyPI :-/ See the requirements.txt file and TODO. My repository: https://hg.python.org/sandbox/benchmarks_perf I would like to push my work as a single giant commit. Brett also proposed me to move the benchmarks repository to GitHub (and so convert it to Git). I don't know if it's appropriate to do all these things at once? What do you think? Reminder: My final goal is to merge again all benchmarks suites (CPython, PyPy, Pyston, Pyjion, ...) into one unique project! Victor

3 years, 10 months

Tracking memory usage

by Victor Stinner

Hi, tl;dr Do you use "perf.py --track_memory"? If yes, for which purpose? Are you using it on Windows or Linux? I'm working on the CPython benchmark suite. It has a --track_memory command option to measure the peak of the memory usage. A main process runs worker processes and track their memory usage. On Linux, the main process reads the "private data" from /proc/pid/smaps of a worker process. It uses a busy-loop: it reads /proc/pid/smaps as fast as possible (with no sleep)! On Windows, PeakPagefileUsage of GetProcessMemoryInfo(process_handle) is used. It uses a loop using a sleep of 1 ms. Do you think that the Linux implementation is reliable? What happens if the worker process only reachs its peak during 1 ms but the main process (the watcher) reads the memory usage every 10 ms? The exact value probably also depends a lot on how the operating system computes the memory usage. RSS is very different from PSS (proportional set size) for example. Linux has also "USS" (unshared memory)... I would prefer to implement the code to track memory in the worker process directly. On Windows, it looks reliable to get the peak after each run. On Linux, it is less clear. Should I use a thread reading /proc/self/smaps in a busy loop? For me, the most reliable option is to use tracemalloc to get the peak of the *Python* memory usage. But this module is only available on Python 3.4 and newer. Another issue is that it slows down a lot the code (something like 2x slower!). I guess that they are two use cases: - read coarse memory usage but don't hit performance - read precise memory usage, ignore performance Victor

3 years, 10 months

perf module 0.7 released

by Victor Stinner

Hi, I released perf 0.7 (quickly followed by a 0.7.1 bugfix): http://perf.readthedocs.io/ I wrote this new version to collect more data in each process. It now reads (and stores) CPUs config, CPUs temperature, CPUs frequency, system load average, etc. Later we can add for example the process RSS peak or other useful metrics. Oh, and the timestamp is now stored per process (run). Again, it's no more global. I noticed a temporarely slowdown which might be caused by a cron task, I'm not sure yet. At least, timestamps should help to debug such issue. I added many CPU metrics because I wanted to analyze why *sometimes* a benchmark suddenly becomes 50% slower (up to 100% slower). It may be related to the CPUs temperature or Intel Turbo Boost, I don't know yet exactly. The previous perf design didn't allow to store information per process, only globally per benchmark. perf 0.7 now supports much better benchmark suites (not only individual benchmarks) and has now a really working --append command. A benchmark file has not enough runs? Run it again with --append! Changes: * new "pybench" command (similar to "python3 -m perf ...") * the --append is now safer and works on benchmark suites * most perf commands now support multiple files and support benchmark suites (not only individual benchmarks) * new dump command and --dump option to display runs * new metadata: cpu_config, cpu_freq, cpu_temp, load_avg_1min In the meanwhile, I also completed and updated my fork of the CPython benchmark suite: https://hg.python.org/sandbox/benchmarks_perf Victor

3 years, 10 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Speed July 2016