[Python-checkins] bpo-37257: obmalloc: stop simple arena thrashing (#14039)

Tim Peters webhook-mailer at python.org
Wed Jun 12 23:41:19 EDT 2019

commit: d1c85a27ea9fe70163cad3443d5e534d94f08284
branch: master
author: Tim Peters <tim.peters at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-06-12T22:41:03-05:00

bpo-37257:  obmalloc:  stop simple arena thrashing (#14039)

GH-14039:  allow (no more than) one wholly empty arena on the usable_arenas list.

This prevents thrashing in some easily-provoked simple cases that could end up creating and destroying an arena on each loop iteration in client code.   Intuitively, if the only arena on the list becomes empty, it makes scant sense to give it back to the system unless we know we'll never need another free pool again before another arena frees a pool.  If the latter obtains, then - yes - this will "waste" an arena.

A Misc/NEWS.d/next/Core and Builtins/2019-06-13-02-27-12.bpo-37257.IMxDvT.rst
M Objects/obmalloc.c

diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-06-13-02-27-12.bpo-37257.IMxDvT.rst b/Misc/NEWS.d/next/Core and Builtins/2019-06-13-02-27-12.bpo-37257.IMxDvT.rst
new file mode 100644
index 000000000000..ac8d90fd2998
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-06-13-02-27-12.bpo-37257.IMxDvT.rst	
@@ -0,0 +1 @@
+Python's small object allocator (``obmalloc.c``) now allows (no more than) one empty arena to remain available for immediate reuse, without returning it to the OS.  This prevents thrashing in simple loops where an arena could be created and destroyed anew on each iteration.
\ No newline at end of file
diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c
index fc7bef619946..622da3ad08fc 100644
--- a/Objects/obmalloc.c
+++ b/Objects/obmalloc.c
@@ -1758,7 +1758,12 @@ pymalloc_free(void *ctx, void *p)
     /* All the rest is arena management.  We just freed
      * a pool, and there are 4 cases for arena mgmt:
      * 1. If all the pools are free, return the arena to
-     *    the system free().
+     *    the system free().  Except if this is the last
+     *    arena in the list, keep it to avoid thrashing:
+     *    keeping one wholly free arena in the list avoids
+     *    pathological cases where a simple loop would
+     *    otherwise provoke needing to allocate and free an
+     *    arena on every iteration.  See bpo-37257.
      * 2. If this is the only free pool in the arena,
      *    add the arena back to the `usable_arenas` list.
      * 3. If the "next" arena has a smaller count of free
@@ -1767,7 +1772,7 @@ pymalloc_free(void *ctx, void *p)
      *    nfreepools.
      * 4. Else there's nothing more to do.
-    if (nf == ao->ntotalpools) {
+    if (nf == ao->ntotalpools && ao->nextarena != NULL) {
         /* Case 1.  First unlink ao from usable_arenas.
         assert(ao->prevarena == NULL ||

More information about the Python-checkins mailing list