[Python-Dev] Problems with the Python Memory Manager

[Josiah Carlson]

Travis Oliphant <oliphant at ee.byu.edu> wrote:
> 
> skip at pobox.com wrote:
> 
> >    Travis> More to the point, however, these scalar objects were allocated
> >    Travis> using the standard PyObject_New and PyObject_Del functions which
> >    Travis> of course use the Python memory manager.  One user ported his
> >    Travis> (long-running) code to the new scipy core and found much to his
> >    Travis> dismay that what used to consume around 100MB now completely
> >    Travis> dominated his machine consuming up to 2GB of memory after only a
> >    Travis> few iterations.  After searching many hours for memory leaks in
> >    Travis> scipy core (not a bad exercise anyway as some were found), the
> >    Travis> real problem was tracked to the fact that his code ended up
> >    Travis> creating and destroying many of these new array scalars.
> >
> >What Python object were his array elements a subclass of?
> 
> These were all scipy core arrays.  The elements were therefore all 
> C-like numbers (floats and integers I think).  If he obtained an element 
> in Python, he would get an instance of a new "array" scalar object which 
> is a builtin extension type written in C.  The important issue though is 
> that these "array" scalars were allocated using PyObject_New and 
> deallocated using PyObject_Del.  The problem is that the Python memory 
> manager did not free the memory. 

This is not a bug, and there doesn't seem to be any plans to change the
behavior: python.org/sf/1338264

If I remember correctly, arrays from the Python standard library (import
array), as well as numarray and Numeric, all store values in their
pure C representations (they don't use PyObject_New unless someone uses
the Python interface to fetch a particular element).  This saves the
overhead of allocating base objects, as well as the 3-5x space blowup
when using Python integers (depending on whether your platform has 32 or
64 bit ints).


> I think definitely, his usage pattern represented a "bad" corner case.  
> An unusable "corner" case in fact.   At any rate, moving to use the 
> system free and malloc fixed the immediate problem.  I mainly wanted to 
> report the problem here just as another piece of anecdotal evidence.

On the one hand, using PyObjects embedded in an array in scientific
Python is a good idea; you can use all of the standard Python
manipulations on them.  On the other hand, other similar projects have
found it more efficient to never embed PyObjects in their arrays, and
just allocate them as necessary on access.

 - Josiah