[Python-Dev] mmap & munmap loop (Was: Compact ordered set

[Victor Stinner]

Any idea why Python calls mmap+munmap more even in a venv?

Victor

Le mer. 27 févr. 2019 à 10:00, INADA Naoki <songofacandy at gmail.com> a écrit :
>
> >
> > > Ah, another interesting point, this huge slowdown happens only when bm_pickle.py
> > > is executed through pyperformance.  When run it directly, slowdown is
> > > not so large.
> >
> > pyperformance runs benchmarks in a virtual environment. I don't know
> > if it has any impact on bm_pickle.
> >
> > Most pyperformance can be run outside a virtual env if required
> > modules are installed on the system. (bm_pickle only requires the
> > stdlib and perf.)
> >
>
> Bingo!
>
> Without venv:
>
> unpickle: Mean +- std dev: 26.9 us +- 0.0 us
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
>  28.78    0.000438           0      1440           read
>  27.33    0.000416           1       440        25 stat
>   9.72    0.000148           1       144           mmap
> ...
>   0.79    0.000012           1        11           munmap
>
> With venv:
>
> % time     seconds  usecs/call     calls    errors syscall
> ------ ----------- ----------- --------- --------- ----------------
>  57.12    0.099023           2     61471           munmap
>  41.87    0.072580           1     61618           mmap
>   0.23    0.000395           1       465        27 stat
>
> unpickle and unpickle_list creates massive same-sized objects, then all objects are
> removed.  If all pools in the arena is freed, munmap is called.
>
> I think we should save some arenas to reuse.  On recent Linux,
> we may be able to use MADV_FREE instead of munmap.
>


-- 
Night gathers, and now my watch begins. It shall not end until my death.