[Python-Dev] Problems with the Python Memory Manager
oliphant at ee.byu.edu
Thu Nov 24 10:54:13 CET 2005
Martin v. Löwis wrote:
> Travis Oliphant wrote:
>> So, I now believe that his code (plus the array scalar extension
>> type) was actually exposing a real bug in the memory manager itself.
>> In theory, the Python memory manager should have been able to re-use
>> the memory for the array-scalar instances because they are always the
>> same size. In practice, the memory was apparently not being re-used
>> but instead new blocks were being allocated to handle the load.
> That is really very hard to believe. Most people on this list would
> probably agree that obmalloc certain *will* reuse deallocated memory
> if the next request is for the very same size (number of bytes) that
> the previously-release object had.
Yes, I see that it does. This became more clear as all the simple tests
I tried failed to reproduce the problem (and I spent some time looking
at the code and reading its comments). I just can't figure out another
explanation for why the problem went away when I went to using the
system malloc other than some kind of corner-case in the Python memory
>> His code is quite complicated and it is difficult to replicate the
> That the code is complex would not so much be a problem: we often
> analyse complex code here. It is a problem that the code is not
> available, and it would be a problem if the problem was not
> reproducable even if you had the code (i.e. if the problem would
> sometimes occur, but not the next day when you ran it again).
The problem was definitely reproducible. On his machine, and on the two
machines I tried to run it on. It without fail rapidly consumed all
> So if you can, please post the code somewhere, and add a bugreport
> on sf.net/projects/python.
I'll try to do this at some point.
I'll have to get permission from him for the actual Python code. The
extension modules he used are all publically available (PyMC). I
changed the memory allocator in scipy --- which eliminated the problem
--- so you'd have to check out an older version of the code from SVN to
see the problem.
Thanks for the tips.
More information about the Python-Dev