Hi Victor,
Fwiw, there is some per-OS (and even apparently
per-Linux-distribution) solution mentioned here:
http://stackoverflow.com/questions/774556/peak-memory-usage-of-a-linux-un...
For me on Arch Linux, "/usr/bin/time -v CMD" returns a reasonable
value in "Maximum resident set size (kbytes)". I guess that on OSes
where this works, it gives a zero-overhead, exact answer.
A bientôt,
Armin.
On Fri, Jul 29, 2016 at 4:42 PM, Victor Stinner
<victor.stinner(a)gmail.com> wrote:
> 2016-07-29 20:51 GMT+02:00 Zachary Ware <zachary.ware+pydev(a)gmail.com>:
>> I think rather than using virtual environments which aren't truly
>> supported by <3.3 anyway, ...
>
> What do you mean? I'm building and destroying dozens of venv everyday
> at work using tox on Python 2.7. The virtualenv command works well,
> no? Do you have issues with it?
Not in particular, just that 3.3+ have official support for venvs
whereas virtualenv is a bit of a hack by necessity. However, the
second point is the real reason I'd rather avoid venvs for this: to
make sure that the interpreters actually use the same exact code, so
that there can't be any setup.py shenanigans that do things
differently between versions/implementations.
--
Zach
2016-07-29 20:51 GMT+02:00 Zachary Ware <zachary.ware+pydev(a)gmail.com>:
> I think rather than using virtual environments which aren't truly
> supported by <3.3 anyway, ...
What do you mean? I'm building and destroying dozens of venv everyday
at work using tox on Python 2.7. The virtualenv command works well,
no? Do you have issues with it?
Victor
On Fri, Jul 29, 2016 at 12:03 PM, Brett Cannon <brett(a)python.org> wrote:
> On Thu, 28 Jul 2016 at 10:25 Victor Stinner <victor.stinner(a)gmail.com>
> wrote:
>> * Should we remove vendor copies of libraries and work with virtual
>> environments? Not all libraries are available on PyPI :-/ See the
>> requirements.txt file and TODO.
>
> If they are not on PyPI then we should just drop the benchmark. And I say we
> do use virtual environments to keep the repo size down.
I think rather than using virtual environments which aren't truly
supported by <3.3 anyway, we should instead make use of pip's
--target, --root, and/or --prefix flags (whatever combination it
takes, I haven't looked into it deeply) to install the packages into a
particular dir which is then added to each benchmarked interpreter's
PYTHONPATH. This way, we're sure that each interpreter is running
exactly the same code.
Either way, I'm for not vendoring libraries. If the library
disappears from PyPI, it's probably not an important workload anymore
anyway.
>> Reminder: My final goal is to merge again all benchmarks suites
>> (CPython, PyPy, Pyston, Pyjion, ...) into one unique project!
>
> I hope this happens!
Me too! I'd also like to get the benchmark runner for
speed.python.org set up to build and benchmark as many interpreters as
possible.
--
Zach
2016-07-29 19:03 GMT+02:00 Brett Cannon <brett(a)python.org>:
>> * Should it support --track_memory? it doesn't support it correctly
>> right now, it's less precise than before (memory is tracked directly
>> in worker processes, no more by the main process) => discuss this
>> point in my previous email
>
> I don't have an opinion as I have never gotten to use the old feature.
As I wrote in my other email, I implemented this feature in perf, so
the benchmark suite will get it for free. The implementation is not
complete, but it's working ;-)
>> * Should we remove vendor copies of libraries and work with virtual
>> environments? Not all libraries are available on PyPI :-/ See the
>> requirements.txt file and TODO.
>
> If they are not on PyPI then we should just drop the benchmark. And I say we
> do use virtual environments to keep the repo size down.
Right. We can start with "a subset" of benchmarks and enlarge the test
suite later, even reimport old benchmark with their dependency not on
PyPI on case by case.
Right now, I only wrote requirements.txt. I didn't touch the Python
which still "hardcodes" PYTHONPATH to get the local copy of
dependencies. It works if you run manually benchmarks in a virtual
environment.
I will write some glue to automate things and make the code "just
work" bfore starting a new thing on GitHub.
Victor
On Thu, 28 Jul 2016 at 10:25 Victor Stinner <victor.stinner(a)gmail.com>
wrote:
> Hi,
>
> I updated all benchmarks of the CPython Benchmark Suite to use my perf
> module. So you get timings of all individual run all *all* benchmarks
> and store them in JSON to analyze them in detail. Each benchmark has a
> full CLI, for example it gets a new --output option to store result as
> JSON directly. But it also gets nice funtions like --hist for
> histogram, --stats for statistics, etc.
>
> The two remaining questions are:
>
> * Should it support --track_memory? it doesn't support it correctly
> right now, it's less precise than before (memory is tracked directly
> in worker processes, no more by the main process) => discuss this
> point in my previous email
>
I don't have an opinion as I have never gotten to use the old feature.
>
> * Should we remove vendor copies of libraries and work with virtual
> environments? Not all libraries are available on PyPI :-/ See the
> requirements.txt file and TODO.
>
If they are not on PyPI then we should just drop the benchmark. And I say
we do use virtual environments to keep the repo size down.
>
> My repository:
> https://hg.python.org/sandbox/benchmarks_perf
>
> I would like to push my work as a single giant commit.
>
> Brett also proposed me to move the benchmarks repository to GitHub
> (and so convert it to Git). I don't know if it's appropriate to do all
> these things at once? What do you think?
>
I say just start a new repo from scratch. There isn't a ton of magical
history in the hg repo that I think we need to have carried around in the
git repo. Plus if we stop shipping project source with the repo then it
will be immensely smaller if we start from scratch.
>
> Reminder: My final goal is to merge again all benchmarks suites
> (CPython, PyPy, Pyston, Pyjion, ...) into one unique project!
I hope this happens!
I modified my perf module to add two new options: --tracemalloc and
--track-memory.
--tracemalloc enables tracemalloc and gets the peak of the traced
Python memory allocations: peak is computed per process.
--track-memory is similar but reads PeakPagefileUsage of
GetProcessMemoryInfo() on Windows or private data from /proc/self/smap
on Linux. The read is done every millisecond (1 ms) in a thread, in
the worker process.
It's not perfect, but it should be "as good" as the "old" CPython
benchmark suite. And it makes the benchmark suite simpler because
tracking memory usage is now done automatically by the perf module.
Victor
Hi,
I updated all benchmarks of the CPython Benchmark Suite to use my perf
module. So you get timings of all individual run all *all* benchmarks
and store them in JSON to analyze them in detail. Each benchmark has a
full CLI, for example it gets a new --output option to store result as
JSON directly. But it also gets nice funtions like --hist for
histogram, --stats for statistics, etc.
The two remaining questions are:
* Should it support --track_memory? it doesn't support it correctly
right now, it's less precise than before (memory is tracked directly
in worker processes, no more by the main process) => discuss this
point in my previous email
* Should we remove vendor copies of libraries and work with virtual
environments? Not all libraries are available on PyPI :-/ See the
requirements.txt file and TODO.
My repository:
https://hg.python.org/sandbox/benchmarks_perf
I would like to push my work as a single giant commit.
Brett also proposed me to move the benchmarks repository to GitHub
(and so convert it to Git). I don't know if it's appropriate to do all
these things at once? What do you think?
Reminder: My final goal is to merge again all benchmarks suites
(CPython, PyPy, Pyston, Pyjion, ...) into one unique project!
Victor
Hi,
tl;dr Do you use "perf.py --track_memory"? If yes, for which purpose?
Are you using it on Windows or Linux?
I'm working on the CPython benchmark suite. It has a --track_memory
command option to measure the peak of the memory usage. A main process
runs worker processes and track their memory usage.
On Linux, the main process reads the "private data" from
/proc/pid/smaps of a worker process. It uses a busy-loop: it reads
/proc/pid/smaps as fast as possible (with no sleep)!
On Windows, PeakPagefileUsage of GetProcessMemoryInfo(process_handle)
is used. It uses a loop using a sleep of 1 ms.
Do you think that the Linux implementation is reliable? What happens
if the worker process only reachs its peak during 1 ms but the main
process (the watcher) reads the memory usage every 10 ms?
The exact value probably also depends a lot on how the operating
system computes the memory usage. RSS is very different from PSS
(proportional set size) for example. Linux has also "USS" (unshared
memory)...
I would prefer to implement the code to track memory in the worker
process directly. On Windows, it looks reliable to get the peak after
each run. On Linux, it is less clear. Should I use a thread reading
/proc/self/smaps in a busy loop?
For me, the most reliable option is to use tracemalloc to get the peak
of the *Python* memory usage. But this module is only available on
Python 3.4 and newer. Another issue is that it slows down a lot the
code (something like 2x slower!).
I guess that they are two use cases:
- read coarse memory usage but don't hit performance
- read precise memory usage, ignore performance
Victor
Hi,
I released perf 0.7 (quickly followed by a 0.7.1 bugfix):
http://perf.readthedocs.io/
I wrote this new version to collect more data in each process. It now
reads (and stores) CPUs config, CPUs temperature, CPUs frequency,
system load average, etc. Later we can add for example the process RSS
peak or other useful metrics.
Oh, and the timestamp is now stored per process (run). Again, it's no
more global. I noticed a temporarely slowdown which might be caused by
a cron task, I'm not sure yet. At least, timestamps should help to
debug such issue.
I added many CPU metrics because I wanted to analyze why *sometimes* a
benchmark suddenly becomes 50% slower (up to 100% slower). It may be
related to the CPUs temperature or Intel Turbo Boost, I don't know yet
exactly.
The previous perf design didn't allow to store information per
process, only globally per benchmark.
perf 0.7 now supports much better benchmark suites (not only
individual benchmarks) and has now a really working --append command.
A benchmark file has not enough runs? Run it again with --append!
Changes:
* new "pybench" command (similar to "python3 -m perf ...")
* the --append is now safer and works on benchmark suites
* most perf commands now support multiple files and support benchmark
suites (not only individual benchmarks)
* new dump command and --dump option to display runs
* new metadata: cpu_config, cpu_freq, cpu_temp, load_avg_1min
In the meanwhile, I also completed and updated my fork of the CPython
benchmark suite:
https://hg.python.org/sandbox/benchmarks_perf
Victor
