[Python-Dev] Problems with the Python Memory Manager
oliphant at ee.byu.edu
Thu Nov 17 00:47:48 CET 2005
Josiah Carlson wrote:
>Robert Kern <robert.kern at gmail.com> wrote:
>> There *is* an array type for general PyObjects in scipy_core, but
>>that's not being used in the code that blows up and has nothing to do
>>with the problem Travis is talking about.
>I seemed to have misunderstood the discussion. Was the original user
>accessing and saving copies of many millions of these doubles?
He *was* accessing them (therefore generating a call to an array-scalar
object creation function). But they *weren't being* saved. They were
being deleted soon after access. That's why it was so confusing that
his memory usage should continue to grow and grow so terribly.
As verified by removing usage of the Python PyObject_MALLOC function, it
was the Python memory manager that was performing poorly. Even though
the array-scalar objects were deleted, the memory manager would not
re-use their memory for later object creation. Instead, the memory
manager kept allocating new arenas to cover the load (when it should
have been able to re-use the old memory that had been freed by the
deleted objects--- again, I don't know enough about the memory manager
to say why this happened).
The fact that it did happen is what I'm reporting on. If nothing will
be done about it (which I can understand), at least this thread might
help somebody else in a similar situation track down why their Python
process consumes all of their memory even though their objects are being
More information about the Python-Dev