[Python-bugs-list] [ python-Bugs-613222 ] memory leaks when importing posix module

[noreply@sourceforge.net]

Bugs item #613222, was opened at 2002-09-23 10:27
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=613222&group_id=5470

Category: Python Interpreter Core
Group: Python 2.2
Status: Open
Resolution: None
Priority: 5
Submitted By: Neal Norwitz (nnorwitz)
>Assigned to: Guido van Rossum (gvanrossum)
Summary: memory leaks when importing posix module

Initial Comment:
The attached program which calls
Py_Initialize/Py_Finalize in a loop demonstrates a
program which grows quite quickly.  This bug effects
2.2.1+ and 2.3.  valgrind reports that the memory is
still reachable, but it seems like a memory leak and
the process definitely grows.

Compile the program with libpython, and run (./mem-test
100000).  Make sure it can import site (which imports
posix module).  While the program is running, do a ps
and watch it grow.  If import site fails, the process
will not grow.

site.py can be as simple as import posix.
I believe the problem is related to
PyStructSequence_Fields for statfs.  But haven't
completely diagnosed the problem.  As I learn more, I
will add comments.

To simply importing or not importing site, mem-test
takes an optional 2nd argument which will
enable/disable loading of site.py.  ./mem-test 100000 1
will prevent import site.

I hope this is understandable, even though the
description wasn't clear.

----------------------------------------------------------------------

>Comment By: Neal Norwitz (nnorwitz)
Date: 2002-09-24 20:37

Message:
Logged In: YES 
user_id=33168

I'm mostly stuck.  I haven't solved the problem, but I do
have some more information.  Attached is a patch to
Object/structseq.c which fixes some problems if memory
allocation fails.  Also, I changed a PyInt_FromLong(1) to
Py_True for __safe_for_unpickling__.  Py_True could also be
used in compile.c::
symtable_load_symbols replacing the variable: implicit.

Some fields which I believe are leaking from the
PyTypeObject are:  tp_members, tp_dict, and tp_bases. 
However, there are more leaks.  I think all the remaining
leaks come from PyType_Ready().  For example, from
add_operators(), PyDescr_NewWrapper().  I thought DECREFing
tp_dict would free these, but it didn't seem to have any effect.

Part of the problem is how should these values be cleaned
up.  Putting cleanup in PyStructSequence_InitType would
guarantee the problem is fixed for all structseqs, but that
doesn't seem like a good idea.  This would assume the
PyTypeObject passed in is initialized.  This is true for
static variables, but not for any other variables.  If
there's a new API for cleanup, all the users of structseq
will need to use it.  Perhaps, this is only an issue in the
core.  I don't know about extension modules.

Finally, I suspect there may be more leaks outside of posix
too.  These seem to mostly come from _Py_ReadyTypes() called
in pythonrun.c::Py_Initialize().

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2002-09-23 12:32

Message:
Logged In: YES 
user_id=6380

This is a good use of your time.  Thanks for looking into
this! Assign back to me when you have a fix for review or a
stumbling block.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=613222&group_id=5470