[Python-Dev] Problems with the Python Memory Manager

[Travis Oliphant]

Josiah Carlson wrote:

>Robert Kern <robert.kern at gmail.com> wrote:
>  
>
>>[1] There *is* an array type for general PyObjects in scipy_core, but
>>that's not being used in the code that blows up and has nothing to do
>>with the problem Travis is talking about.
>>    
>>
>
>I seemed to have misunderstood the discussion.  Was the original user
>accessing and saving copies of many millions of these doubles?  
>
He *was* accessing them (therefore generating a call to an array-scalar 
object creation function).  But they *weren't being* saved.  They were 
being deleted soon after access.   That's why it was so confusing that 
his memory usage should continue to grow and grow so terribly.

As verified by removing usage of the Python PyObject_MALLOC function, it 
was the Python memory manager that was performing poorly.   Even though 
the array-scalar objects were deleted, the memory manager would not 
re-use their memory for later object creation. Instead, the memory 
manager kept allocating new arenas to cover the load (when it should 
have been able to re-use the old memory that had been freed by the 
deleted objects--- again, I don't know enough about the memory manager 
to say why this happened).

The fact that it did happen is what I'm reporting on.  If nothing will 
be done about it (which I can understand), at least this thread might 
help somebody else in a similar situation track down why their Python 
process consumes all of their memory even though their objects are being 
deleted appropriately.

Best,

-Travis