[Python-bugs-list] [ python-Bugs-613222 ] memory leaks when importing posix module

noreply@sourceforge.net noreply@sourceforge.net
Thu, 26 Dec 2002 07:16:56 -0800


Bugs item #613222, was opened at 2002-09-23 10:27
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=613222&group_id=5470

Category: Python Interpreter Core
Group: Python 2.2
Status: Open
Resolution: None
>Priority: 3
Submitted By: Neal Norwitz (nnorwitz)
Assigned to: Guido van Rossum (gvanrossum)
Summary: memory leaks when importing posix module

Initial Comment:
The attached program which calls
Py_Initialize/Py_Finalize in a loop demonstrates a
program which grows quite quickly.  This bug effects
2.2.1+ and 2.3.  valgrind reports that the memory is
still reachable, but it seems like a memory leak and
the process definitely grows.

Compile the program with libpython, and run (./mem-test
100000).  Make sure it can import site (which imports
posix module).  While the program is running, do a ps
and watch it grow.  If import site fails, the process
will not grow.

site.py can be as simple as import posix.
I believe the problem is related to
PyStructSequence_Fields for statfs.  But haven't
completely diagnosed the problem.  As I learn more, I
will add comments.

To simply importing or not importing site, mem-test
takes an optional 2nd argument which will
enable/disable loading of site.py.  ./mem-test 100000 1
will prevent import site.

I hope this is understandable, even though the
description wasn't clear.

----------------------------------------------------------------------

>Comment By: Guido van Rossum (gvanrossum)
Date: 2002-12-26 10:16

Message:
Logged In: YES 
user_id=6380

I expect I will have no time to look into this further any
time soon. But if you have fixes that clearly fix leaks,
please check them in!

(Shouldn't this be in the 2.3 group?)


----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2002-09-24 20:37

Message:
Logged In: YES 
user_id=33168

I'm mostly stuck.  I haven't solved the problem, but I do
have some more information.  Attached is a patch to
Object/structseq.c which fixes some problems if memory
allocation fails.  Also, I changed a PyInt_FromLong(1) to
Py_True for __safe_for_unpickling__.  Py_True could also be
used in compile.c::
symtable_load_symbols replacing the variable: implicit.

Some fields which I believe are leaking from the
PyTypeObject are:  tp_members, tp_dict, and tp_bases. 
However, there are more leaks.  I think all the remaining
leaks come from PyType_Ready().  For example, from
add_operators(), PyDescr_NewWrapper().  I thought DECREFing
tp_dict would free these, but it didn't seem to have any effect.

Part of the problem is how should these values be cleaned
up.  Putting cleanup in PyStructSequence_InitType would
guarantee the problem is fixed for all structseqs, but that
doesn't seem like a good idea.  This would assume the
PyTypeObject passed in is initialized.  This is true for
static variables, but not for any other variables.  If
there's a new API for cleanup, all the users of structseq
will need to use it.  Perhaps, this is only an issue in the
core.  I don't know about extension modules.

Finally, I suspect there may be more leaks outside of posix
too.  These seem to mostly come from _Py_ReadyTypes() called
in pythonrun.c::Py_Initialize().

----------------------------------------------------------------------

Comment By: Guido van Rossum (gvanrossum)
Date: 2002-09-23 12:32

Message:
Logged In: YES 
user_id=6380

This is a good use of your time.  Thanks for looking into
this! Assign back to me when you have a fix for review or a
stumbling block.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=613222&group_id=5470