[Python-Dev] Problems with the Python Memory Manager
oliphant at ee.byu.edu
Thu Nov 17 03:15:04 CET 2005
Jim Jewett wrote:
>Do you have the code that caused problems?
Yes. I was able to reproduce his trouble and was trying to debug it.
>The things I would check first are
>(1) Is he allocating (peak usage) a type (such as integers) that
>never gets returned to the free pool, in case you need more of that
No, I don't think so.
>(2) Is he allocating new _types_, which I think don't get properly
Bingo. Yes, definitely allocating new _types_ (an awful lot of them...)
--- that's what the "array scalars" are: new types created in C. If
they don't get properly collected then that would definitely have
created the problem. It would seem this should be advertised when
telling people to use PyObject_New for allocating new memory for an object.
>(3) Is there something in his code that keeps a live reference, or at
>least a spotty memory usage so that the memory can't be cleanly
No, that's where I thought the problem was, at first. I spent a lot of
time tracking down references. What finally convinced me it was the
Python memory manager was when I re-wrote the tp->alloc functions of the
new types to use the system malloc instead of PyObject_Malloc. As
soon as I did this the problems disappeared and memory stayed constant.
Thanks for your comments,
More information about the Python-Dev