[Python-Dev] Problems with the Python Memory Manager
oliphant at ee.byu.edu
Wed Nov 16 19:20:48 CET 2005
skip at pobox.com wrote:
> Travis> More to the point, however, these scalar objects were allocated
> Travis> using the standard PyObject_New and PyObject_Del functions which
> Travis> of course use the Python memory manager. One user ported his
> Travis> (long-running) code to the new scipy core and found much to his
> Travis> dismay that what used to consume around 100MB now completely
> Travis> dominated his machine consuming up to 2GB of memory after only a
> Travis> few iterations. After searching many hours for memory leaks in
> Travis> scipy core (not a bad exercise anyway as some were found), the
> Travis> real problem was tracked to the fact that his code ended up
> Travis> creating and destroying many of these new array scalars.
>What Python object were his array elements a subclass of?
These were all scipy core arrays. The elements were therefore all
C-like numbers (floats and integers I think). If he obtained an element
in Python, he would get an instance of a new "array" scalar object which
is a builtin extension type written in C. The important issue though is
that these "array" scalars were allocated using PyObject_New and
deallocated using PyObject_Del. The problem is that the Python memory
manager did not free the memory.
> Travis> In the long term, what is the status of plans to re-work the
> Travis> Python Memory manager to free memory that it acquires (or
> Travis> improve the detection of already freed memory locations).
>None that I'm aware of. It's seen a great deal of work in the past and
>generally doesn't cause problems. Maybe your user's usage patterns were
>a bad corner case. It's hard to tell without more details.
I think definitely, his usage pattern represented a "bad" corner case.
An unusable "corner" case in fact. At any rate, moving to use the
system free and malloc fixed the immediate problem. I mainly wanted to
report the problem here just as another piece of anecdotal evidence.
More information about the Python-Dev