[Python-Dev] Problems with the Python Memory Manager

[Travis Oliphant]

Jim Jewett wrote:

>Do you have the code that caused problems?
>  
>
Yes.  I was able to reproduce his trouble and was trying to debug it.

>The things I would check first are
>
>(1)  Is he allocating (peak usage) a type (such as integers) that
>never gets returned to the free pool, in case you need more of that
>same type?
>  
>
No, I don't think so.

>(2)  Is he allocating new _types_, which I think don't get properly
>
> collected.
>  
>

Bingo.  Yes, definitely allocating new _types_ (an awful lot of them...) 
--- that's what the "array scalars" are: new types created in C.  If 
they don't get properly collected then that would definitely have 
created the problem.  It would seem this should be advertised when 
telling people to use PyObject_New for allocating new memory for an object.

>(3)  Is there something in his code that keeps a live reference, or at
>least a spotty memory usage so that the memory can't be cleanly
>released?
>
>  
>
No, that's where I thought the problem was, at first.  I spent a lot of 
time tracking down references.    What finally convinced me it was the 
Python memory manager was when I re-wrote the tp->alloc functions of the 
new types to use the system malloc instead of PyObject_Malloc.    As 
soon as I did this the problems disappeared and memory stayed constant. 

Thanks for your comments,

-Travis