[Python-Dev] Problems with the Python Memory Manager

[Travis Oliphant]

Martin v. Löwis wrote:

> Travis Oliphant wrote:
>
>> So, I now believe that his code (plus the array scalar extension 
>> type) was actually exposing a real bug in the memory manager itself.  
>> In theory, the Python memory manager should have been able to re-use 
>> the memory for the array-scalar instances because they are always the 
>> same size.  In practice, the memory was apparently not being re-used 
>> but instead new blocks were being allocated to handle the load.
>
>
> That is really very hard to believe. Most people on this list would
> probably agree that obmalloc certain *will* reuse deallocated memory
> if the next request is for the very same size (number of bytes) that
> the previously-release object had.


Yes, I see that it does.  This became more clear as all the simple tests 
I tried failed to reproduce the problem (and I spent some time looking 
at the code and reading its comments).   I just can't figure out another 
explanation for why the problem went away when I went to using the 
system malloc other than some kind of corner-case in the Python memory 
allocator.

>
>> His code is quite complicated and it is difficult to replicate the 
>> problem.  
>
>
> That the code is complex would not so much be a problem: we often
> analyse complex code here. It is a problem that the code is not
> available, and it would be a problem if the problem was not
> reproducable even if you had the code (i.e. if the problem would
> sometimes occur, but not the next day when you ran it again).
>
The problem was definitely reproducible.  On his machine, and on the two 
machines I tried to run it on.  It without fail rapidly consumed all 
available memory.

> So if you can, please post the code somewhere, and add a bugreport
> on sf.net/projects/python.
>
I'll try to do this at some point. 

I'll have to get permission from him for the actual Python code.  The 
extension modules he used are all publically available (PyMC).  I 
changed the memory allocator in scipy --- which eliminated the problem 
--- so you'd have to check out an older version of the code from SVN to 
see the problem.

Thanks for the tips.

-Travis