[Python-Dev] Problems with the Python Memory Manager

[Travis Oliphant]

I know (thanks to Google) that much has been said in the past about the 
Python Memory Manager.  My purpose in posting is simply to given a 
use-case example of how the current memory manager (in Python 2.4.X) can 
be problematic in scientific/engineering code.

Scipy core is a replacement for Numeric.  One of the things scipy core 
does is define a new python scalar object for ever data type that an 
array can have (currently 21).   This has many advantages and is made 
feasible by the ability of Python to subtype in C.   These scalars all 
inherit from the standard Python types where there is a correspondence.

More to the point, however, these scalar objects were allocated using 
the standard PyObject_New and PyObject_Del functions which of course use 
the Python memory manager.    One user ported his (long-running) code to 
the new scipy core and found much to his dismay that what used to 
consume around 100MB now completely dominated his machine consuming up 
to 2GB of memory after only a few iterations.  After searching many 
hours for memory leaks in scipy core (not a bad exercise anyway as some 
were found), the real problem was tracked to the fact that his code 
ended up creating and destroying many of these new array scalars.  

The Python memory manager was not reusing memory (even though 
PyObject_Del was being called).  I don't know enough about the memory 
manager to understand why that was happening.  However, changing the 
allocation from PyObject_New to malloc and from PyObject_Del to free, 
fixed the problems this user was seeing.   Now the code runs for a long 
time consuming only around 100MB at-a-time.

Thus, all of the objects in scipy core now use system malloc and system 
free for their memory needs.   Perhaps this is unfortunate, but it was 
the only solution I could see in the short term.

In the long term, what is the status of plans to re-work the Python 
Memory manager to free memory that it acquires (or improve the detection 
of already freed memory locations).  I see from other postings that this 
has been a problem for other people as well.   Also, is there a 
recommended way for dealing with this problem other than using system 
malloc and system free (or I suppose writing your own specialized memory 
manager).

Thanks for any feedback,


-Travis Oliphant