[Python-Dev] mmap & munmap loop (Was: Compact ordered set

[INADA Naoki]

On Wed, Feb 27, 2019 at 9:59 PM Victor Stinner <vstinner at redhat.com> wrote:

> Maybe pickle is inefficient in its memory management and causes a lot
> of memory fragmentation?
>

No, it is not relating to pickle efficiency and memory fragmentation.
This problem is happened because pymalloc doesn't have no hysteresis
between map & unmap arenas.
Any workload creating some objects and release them soon may affected
by this problem.

When there are no free pool, pymalloc use  mmap to create new arena (256KB).
pymalloc allocates new pools (= pages) from the arena.  It cause minor
fault.
Linux allocates real memory to the page and RSS is increased.

Then, when all objects newly created in the pool are destroyed, all pools
in the
arena are free.  pymalloc calls munmap() soon.

unpickle can be affected by this problem than pure Python because:

* unpickle is creating Python objects quickly.  fault overhead is
relatively large.
* Python code may creates junk memory block (e.g. cached frame object,
freeslot, etc)
  but C pickle code doesn't creates such junks.  So newly allocated pools
are
  freed very easily.

I think this issue can be avoided easily.  When arena is empty but the arena
is head of usable_arenas, don't call munmap for it.
I confirmed m2.py can't reproduce the issue with this patch.

diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c
index 1c2a32050f..a19b3aca06 100644
--- a/Objects/obmalloc.c
+++ b/Objects/obmalloc.c
@@ -1672,7 +1672,7 @@ pymalloc_free(void *ctx, void *p)
      *    nfreepools.
      * 4. Else there's nothing more to do.
      */
-    if (nf == ao->ntotalpools) {
+    if (nf == ao->ntotalpools && ao != usable_arenas) {
         /* Case 1.  First unlink ao from usable_arenas.
          */
         assert(ao->prevarena == NULL ||

-- 
INADA Naoki  <songofacandy at gmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20190227/8585061b/attachment-0001.html>