[New-bugs-announce] [issue37146] opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting

[STINNER Victor]

New submission from STINNER Victor <vstinner at redhat.com>:

opcode cache for LOAD_GLOBAL introduced false alarm in memory leak hunting (python3 -m test -R 3:3 ...).

=> opcache: bpo-26219.


Before the change:

$ git checkout 91234a16367b56ca03ee289f7c03a34d4cfec4c8^
$ make && ./python -m test -R 3:3 test_pprint 
...
Tests result: SUCCESS

After the change:

$ git checkout 91234a16367b56ca03ee289f7c03a34d4cfec4c8
$ make && ./python -m test -R 3:3 test_pprint 
...
test_pprint leaked [4, 2, 4] memory blocks, sum=10
...

The problem is that at each iteration of regrtest -R 3:3 (6 iterations), a few more code objects get this opcache allocated.

There are different solutions to fix regrtest -R 3:3.


(*) Always optimize:

diff --git a/Python/ceval.c b/Python/ceval.c
index 411ba3d73c..6cd148efba 100644
--- a/Python/ceval.c
+++ b/Python/ceval.c
@@ -103,7 +103,7 @@ static long dxp[256];
 #endif
 
 /* per opcode cache */
-#define OPCACHE_MIN_RUNS 1024  /* create opcache when code executed this time */
+#define OPCACHE_MIN_RUNS 1  /* create opcache when code executed this time */
 #define OPCACHE_STATS 0  /* Enable stats */
 
 #if OPCACHE_STATS

$ make && ./python -m test -R 3:3 test_pprint 
...
Tests result: SUCCESS


(*) Never optimmize: disable opcache until a better fix can be found

diff --git a/Python/ceval.c b/Python/ceval.c
index 411ba3d73c..3c85df6fea 100644
--- a/Python/ceval.c
+++ b/Python/ceval.c
@@ -1230,6 +1230,7 @@ _PyEval_EvalFrameDefault(PyFrameObject *f, int throwflag)
     f->f_stacktop = NULL;       /* remains NULL unless yield suspends frame */
     f->f_executing = 1;
 
+#if 0
     if (co->co_opcache_flag < OPCACHE_MIN_RUNS) {
         co->co_opcache_flag++;
         if (co->co_opcache_flag == OPCACHE_MIN_RUNS) {
@@ -1244,6 +1245,7 @@ _PyEval_EvalFrameDefault(PyFrameObject *f, int throwflag)
 #endif
         }
     }
+#endif
 
 #ifdef LLTRACE
     lltrace = _PyDict_GetItemId(f->f_globals, &PyId___ltrace__) != NULL;

$ make && ./python -m test -R 3:3 test_pprint 
...
Tests result: SUCCESS


(*) Find a way to explicitly deoptimize all code objects

Modules/gcmodule.c has a clear_freelists() function called by collect() if generation == NUM_GENERATIONS-1: on when gc.collect() is collected explicitly for example.

Lib/test/libregrtest/refleak.py also has a dash_R_cleanup() function which clears many caches.

Problem: currently, code objects are not explicitly tracked (for example, they are not tracked in a double linked list).


(*) Add way more warmup iterations to regrtest in buildbots.

I dislike this option. A build on a refleak buildbot worker already takes 2 to 3 hours. Adding more warmup would make a build even way more slower.

----------
components: Interpreter Core
messages: 344469
nosy: inada.naoki, vstinner, yselivanov
priority: normal
severity: normal
status: open
title: opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting
versions: Python 3.8

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue37146>
_______________________________________