[Python-Dev] Problems with the Python Memory Manager

[Travis Oliphant]

skip at pobox.com wrote:

>    Travis> More to the point, however, these scalar objects were allocated
>    Travis> using the standard PyObject_New and PyObject_Del functions which
>    Travis> of course use the Python memory manager.  One user ported his
>    Travis> (long-running) code to the new scipy core and found much to his
>    Travis> dismay that what used to consume around 100MB now completely
>    Travis> dominated his machine consuming up to 2GB of memory after only a
>    Travis> few iterations.  After searching many hours for memory leaks in
>    Travis> scipy core (not a bad exercise anyway as some were found), the
>    Travis> real problem was tracked to the fact that his code ended up
>    Travis> creating and destroying many of these new array scalars.
>
>What Python object were his array elements a subclass of?
>  
>
These were all scipy core arrays.  The elements were therefore all 
C-like numbers (floats and integers I think).  If he obtained an element 
in Python, he would get an instance of a new "array" scalar object which 
is a builtin extension type written in C.  The important issue though is 
that these "array" scalars were allocated using PyObject_New and 
deallocated using PyObject_Del.  The problem is that the Python memory 
manager did not free the memory. 

>    Travis> In the long term, what is the status of plans to re-work the
>    Travis> Python Memory manager to free memory that it acquires (or
>    Travis> improve the detection of already freed memory locations).  
>
>None that I'm aware of.  It's seen a great deal of work in the past and
>generally doesn't cause problems.  Maybe your user's usage patterns were
>a bad corner case.  It's hard to tell without more details.
>  
>
I think definitely, his usage pattern represented a "bad" corner case.  
An unusable "corner" case in fact.   At any rate, moving to use the 
system free and malloc fixed the immediate problem.  I mainly wanted to 
report the problem here just as another piece of anecdotal evidence.

-Travis