[Python-Dev] Problems with the Python Memory Manager

[Travis Oliphant]

Martin v. Löwis wrote:

> Travis Oliphant wrote:
>
>> As verified by removing usage of the Python PyObject_MALLOC function, 
>> it was the Python memory manager that was performing poorly.   Even 
>> though the array-scalar objects were deleted, the memory manager 
>> would not re-use their memory for later object creation. Instead, the 
>> memory manager kept allocating new arenas to cover the load (when it 
>> should have been able to re-use the old memory that had been freed by 
>> the deleted objects--- again, I don't know enough about the memory 
>> manager to say why this happened).
>
>
> One way (I think the only way) this could happen if:
> - the objects being allocated are all smaller than 256 bytes
> - when allocating new objects, the requested size was different
>   from any other size previously deallocated.


In one version of the code I had moved all objects from the Python 
memory manager to the system malloc *except* the array scalars.   The 
problem still remained, so I'm pretty sure these were the problem.    
The array scalars are all less than 256 bytes but they are always the 
same number of bytes. 

>
> So if you first allocate 1,000,000 objects of size 200, and then
> release them, and then allocate 1,000,000 objects of size 208,
> the memory is not reused.

That is useful information.   I don't think his code was doing that kind 
of thing, but it definitely provides something to check on.

Previously I was using the standard tp_alloc and tp_free methods (I was 
not setting them but just letting PyType_Ready fill those slots in with 
the default values).    When I changed these methods to ones that used 
system free and system malloc the problem went away.  That's why I 
attribute the issue to the Python memory manager.   Of course, it's 
always possible that I was doing something wrong, but I really did try 
to make sure I wasn't making a mistake.  I didn't do anything fancy with 
the Python memory allocator. 

The array scalars all subclass from each other in C, though.  I don't 
see how that could be relevant, but I could be missing something.

-Travis