[Python-Dev] mmap & munmap loop (Was: Compact ordered set

INADA Naoki songofacandy at gmail.com
Wed Feb 27 03:59:57 EST 2019


>
> > Ah, another interesting point, this huge slowdown happens only when
bm_pickle.py
> > is executed through pyperformance.  When run it directly, slowdown is
> > not so large.
>
> pyperformance runs benchmarks in a virtual environment. I don't know
> if it has any impact on bm_pickle.
>
> Most pyperformance can be run outside a virtual env if required
> modules are installed on the system. (bm_pickle only requires the
> stdlib and perf.)
>

Bingo!

Without venv:

unpickle: Mean +- std dev: 26.9 us +- 0.0 us
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 28.78    0.000438           0      1440           read
 27.33    0.000416           1       440        25 stat
  9.72    0.000148           1       144           mmap
...
  0.79    0.000012           1        11           munmap

With venv:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 57.12    0.099023           2     61471           munmap
 41.87    0.072580           1     61618           mmap
  0.23    0.000395           1       465        27 stat

unpickle and unpickle_list creates massive same-sized objects, then all
objects are
removed.  If all pools in the arena is freed, munmap is called.

I think we should save some arenas to reuse.  On recent Linux,
we may be able to use MADV_FREE instead of munmap.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190227/b0c5b6d7/attachment.html>


More information about the Python-Dev mailing list