[Python-Dev] Problems with the Python Memory Manager

Robert Kern robert.kern at gmail.com
Wed Nov 16 21:41:00 CET 2005

Josiah Carlson wrote:
> Travis Oliphant <oliphant at ee.byu.edu> wrote:

>>I think definitely, his usage pattern represented a "bad" corner case.  
>>An unusable "corner" case in fact.   At any rate, moving to use the 
>>system free and malloc fixed the immediate problem.  I mainly wanted to 
>>report the problem here just as another piece of anecdotal evidence.
> On the one hand, using PyObjects embedded in an array in scientific
> Python is a good idea; you can use all of the standard Python
> manipulations on them.  On the other hand, other similar projects have
> found it more efficient to never embed PyObjects in their arrays, and
> just allocate them as necessary on access.

That's not what we're doing[1]. The scipy_core arrays here are just
blocks of C doubles. However, the offending code (I believe Chris
Fonnesbeck's PyMC, but I could be mistaken) frequently indexes into
these arrays to get scalar values. In scipy_core, we've defined a set of
numerical types that generally behave like Python ints and floats but
have the underlying storage of the appropriate C data type and have the
various array attributes and methods. When the result of an indexing
operation is a scalar (e.g., arange(10)[0]), it always returns an
instance of the appropriate scalar type. We are "just allocat[ing] them
as necessary on access."

[1] There *is* an array type for general PyObjects in scipy_core, but
that's not being used in the code that blows up and has nothing to do
with the problem Travis is talking about.

Robert Kern
robert.kern at gmail.com

"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter

More information about the Python-Dev mailing list