Re: [Python-Dev] Problems with the Python Memory Manager
Jim Jewett wrote:
Do you have the code that caused problems?
Yes. I was able to reproduce his trouble and was trying to debug it.
The things I would check first are
(1) Is he allocating (peak usage) a type (such as integers) that never gets returned to the free pool, in case you need more of that same type?
No, I don't think so.
(2) Is he allocating new _types_, which I think don't get properly
collected.
Bingo. Yes, definitely allocating new _types_ (an awful lot of them...) --- that's what the "array scalars" are: new types created in C. If they don't get properly collected then that would definitely have created the problem. It would seem this should be advertised when telling people to use PyObject_New for allocating new memory for an object.
(3) Is there something in his code that keeps a live reference, or at least a spotty memory usage so that the memory can't be cleanly released?
No, that's where I thought the problem was, at first. I spent a lot of time tracking down references. What finally convinced me it was the Python memory manager was when I re-wrote the tp->alloc functions of the new types to use the system malloc instead of PyObject_Malloc. As soon as I did this the problems disappeared and memory stayed constant. Thanks for your comments, -Travis
On 17-nov-2005, at 3:15, Travis Oliphant wrote:
Jim Jewett wrote:
(2) Is he allocating new _types_, which I think don't get properly
collected.
Bingo. Yes, definitely allocating new _types_ (an awful lot of them...) --- that's what the "array scalars" are: new types created in C.
Do you really mean that someArray[1] will create a new type to represent the second element of someArray? I would guess that you create an instance of a type defined in your extension. Ronald
Bingo. Yes, definitely allocating new _types_ (an awful lot of them...) --- that's what the "array scalars" are: new types created in C.
Do you really mean that someArray[1] will create a new type to represent the second element of someArray? I would guess that you create an instance of a type defined in your extension.
O.K. my bad. I can see that I was confusing in my recent description and possibly misunderstood the questions I was asked. It can get confusing given the dynamic nature of Python. The array scalars are new statically defined (in C) types (just like regular Python integers and regular Python floats). The ndarray is also a statically defined type. The ndarray holds raw memory interpreted in a certain fashion (very similar to Python's array module). Each ndarray can have a certain data type. For every data type that an array can be, there is a corresponding "array scalar" type. All of these are statically defined types. We are only talking about instances of these defined types. When the result of a user operation with an ndarray is a scalar, an instance of the appropriate "array scalar" type is created and passed back to the user. Previously we were using PyObject_New in the tp_alloc slot and PyObject_Del in the tp_free slot of the typeobject structure in order to create and destroy the memory for these instances. In this particular application, the user ended up creating many, many instances of these array scalars and then deleting them soon after. Despite the fact that he was not retaining any references to these scalars (PyObject_Del had been called on them), his application crawled to a halt after only several hunderd iterations consuming all of the available system memory. To verify that indeed no references were being kept, I did a detailed analysis of the result of sys.getobjects() using a debug build of Python. When I replaced PyObject_New (with malloc and PyObject_Init) and PyObject_Del (with free) for the "array scalars" types in scipy core, the users memory problems magically disappeared. I therefore assume that the problem is the memory manager in Python. Initially, I thought this was the old problem of Python not freeing memory once it grabs it. But, that should not have been a problem here, because the code quickly frees most of the objects it creates and so Python should have been able to re-use the memory. So, I now believe that his code (plus the array scalar extension type) was actually exposing a real bug in the memory manager itself. In theory, the Python memory manager should have been able to re-use the memory for the array-scalar instances because they are always the same size. In practice, the memory was apparently not being re-used but instead new blocks were being allocated to handle the load. His code is quite complicated and it is difficult to replicate the problem. I realize this is not helpful for fixing the Python memory manager, and I wish I could be more helpful. However, replacing PyObject_New with malloc does solve the problem for us and that may help anybody else in this situation in the future. Best regards, -Travis
Travis Oliphant wrote:
So, I now believe that his code (plus the array scalar extension type) was actually exposing a real bug in the memory manager itself. In theory, the Python memory manager should have been able to re-use the memory for the array-scalar instances because they are always the same size. In practice, the memory was apparently not being re-used but instead new blocks were being allocated to handle the load.
That is really very hard to believe. Most people on this list would probably agree that obmalloc certain *will* reuse deallocated memory if the next request is for the very same size (number of bytes) that the previously-release object had.
His code is quite complicated and it is difficult to replicate the problem.
That the code is complex would not so much be a problem: we often analyse complex code here. It is a problem that the code is not available, and it would be a problem if the problem was not reproducable even if you had the code (i.e. if the problem would sometimes occur, but not the next day when you ran it again). So if you can, please post the code somewhere, and add a bugreport on sf.net/projects/python. Regards, Martin
Martin v. Löwis wrote:
Travis Oliphant wrote:
So, I now believe that his code (plus the array scalar extension type) was actually exposing a real bug in the memory manager itself. In theory, the Python memory manager should have been able to re-use the memory for the array-scalar instances because they are always the same size. In practice, the memory was apparently not being re-used but instead new blocks were being allocated to handle the load.
That is really very hard to believe. Most people on this list would probably agree that obmalloc certain *will* reuse deallocated memory if the next request is for the very same size (number of bytes) that the previously-release object had.
Yes, I see that it does. This became more clear as all the simple tests I tried failed to reproduce the problem (and I spent some time looking at the code and reading its comments). I just can't figure out another explanation for why the problem went away when I went to using the system malloc other than some kind of corner-case in the Python memory allocator.
His code is quite complicated and it is difficult to replicate the problem.
That the code is complex would not so much be a problem: we often analyse complex code here. It is a problem that the code is not available, and it would be a problem if the problem was not reproducable even if you had the code (i.e. if the problem would sometimes occur, but not the next day when you ran it again).
The problem was definitely reproducible. On his machine, and on the two machines I tried to run it on. It without fail rapidly consumed all available memory.
So if you can, please post the code somewhere, and add a bugreport on sf.net/projects/python.
I'll try to do this at some point. I'll have to get permission from him for the actual Python code. The extension modules he used are all publically available (PyMC). I changed the memory allocator in scipy --- which eliminated the problem --- so you'd have to check out an older version of the code from SVN to see the problem. Thanks for the tips. -Travis
Martin v. Löwis wrote:
That the code is complex would not so much be a problem: we often analyse complex code here. It is a problem that the code is not available, and it would be a problem if the problem was not reproducable even if you had the code (i.e. if the problem would sometimes occur, but not the next day when you ran it again).
You can get the version of scipy_core just before the fix that Travis applied: svn co -r 1488 http://svn.scipy.org/svn/scipy_core/trunk The fix: http://projects.scipy.org/scipy/scipy_core/changeset/1489 http://projects.scipy.org/scipy/scipy_core/changeset/1490 Here's some code that eats up memory with rev1488, but not with the HEAD: """ import scipy a = scipy.arange(10) for i in xrange(10000000): x = a[5] """ -- Robert Kern robert.kern@gmail.com "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter
Hi, On Thu, Nov 24, 2005 at 01:59:57AM -0800, Robert Kern wrote:
You can get the version of scipy_core just before the fix that Travis applied:
Now we can start debugging :-)
This changeset alone fixes the small example you provided. However, compiling python "--without-pymalloc" doesn't fix it, so we can't blame the memory allocator. That's all I can say; I am rather clueless as to how the above patch manages to make any difference even without pymalloc. A bientot, Armin
Hi, Ok, here is the reason for the leak... There is in scipy a type called 'int32_arrtype' which inherits from both another scipy type called 'signedinteger_arrtype', and from 'int'. Obscure! This is not 100% officially allowed: you are inheriting from two C types. You're living dangerously! Now in this case it mostly works as expected, because the parent scipy type has no field at all, so it's mostly like inheriting from both 'object' and 'int' -- which is allowed, or would be if the bases were written in the opposite order. But still, something confuses the fragile logic of typeobject.c. (I'll leave this bit to scipy people to debug :-) The net result is that unless you force your own tp_free as in revision 1490, the type 'int32_arrtype' has tp_free set to int_free(), which is the normal tp_free of 'int' objects. This causes all deallocated int32_arrtype instances to be added to the CPython free list of integers instead of being freed! A bientot, Armin
Armin Rigo wrote:
Hi,
Ok, here is the reason for the leak...
There is in scipy a type called 'int32_arrtype' which inherits from both another scipy type called 'signedinteger_arrtype', and from 'int'. Obscure! This is not 100% officially allowed: you are inheriting from two C types. You're living dangerously!
This is allowed because the two types have compatible binaries (in fact the signed integer type is only the PyObject_HEAD)
Now in this case it mostly works as expected, because the parent scipy type has no field at all, so it's mostly like inheriting from both 'object' and 'int' -- which is allowed, or would be if the bases were written in the opposite order. But still, something confuses the fragile logic of typeobject.c. (I'll leave this bit to scipy people to debug :-)
This is definitely possible. I've tripped up in this logic before. I was beginning to suspect that it might have something to do with what is going on.
The net result is that unless you force your own tp_free as in revision 1490, the type 'int32_arrtype' has tp_free set to int_free(), which is the normal tp_free of 'int' objects. This causes all deallocated int32_arrtype instances to be added to the CPython free list of integers instead of being freed!
I'm not sure this is true, It sounds plausible but I will have to check. Previously the tp_free should have been inherited as PyObject_Del for the int32_arrtype. Unless the typeobject.c code copied the tp_free from the wrong base type, this shouldn't have been the case. Thanks for the pointers. It sounds like we're getting close. Perhaps the problem is in typeobject.c .... -Travis
Armin Rigo wrote:
Hi,
Ok, here is the reason for the leak...
There is in scipy a type called 'int32_arrtype' which inherits from both another scipy type called 'signedinteger_arrtype', and from 'int'. Obscure! This is not 100% officially allowed: you are inheriting from two C types. You're living dangerously!
Now in this case it mostly works as expected, because the parent scipy type has no field at all, so it's mostly like inheriting from both 'object' and 'int' -- which is allowed, or would be if the bases were written in the opposite order. But still, something confuses the fragile logic of typeobject.c. (I'll leave this bit to scipy people to debug :-)
The net result is that unless you force your own tp_free as in revision 1490, the type 'int32_arrtype' has tp_free set to int_free(), which is the normal tp_free of 'int' objects. This causes all deallocated int32_arrtype instances to be added to the CPython free list of integers instead of being freed!
I can confirm that indeed the int32_arrtype object gets the tp_free slot from it's second parent (the python integer type) instead of its first parent (the new, empty signed integer type). I just did a printf after PyType_Ready was called to see what the tp_free slot contained, and indeed it contained the wrong thing. I suspect this may also be true of the float64_arrtype as well (which inherits from Python's float type). What I don't understand is why the tp_free slot from the second base type got copied over into the tp_free slot of the child. It should have received the tp_free slot of the first parent, right? I'm still looking for why that would be the case. I think, though, Armin has identified the real culprit of the problem. I apologize for any consternation over the memory manager that may have taken place. This problem is obviously an issue of dual inheritance in C. I understand this is not well tested code, but in principle it should work correctly, right? I'll keep looking to see if I made a mistake in believing that the int32_arrtype should have inherited its tp_free slot from the first parent and not the second. -Travis
Armin Rigo wrote:
Hi,
Ok, here is the reason for the leak...
There is in scipy a type called 'int32_arrtype' which inherits from both another scipy type called 'signedinteger_arrtype', and from 'int'. Obscure! This is not 100% officially allowed: you are inheriting from two C types. You're living dangerously!
Now in this case it mostly works as expected, because the parent scipy type has no field at all, so it's mostly like inheriting from both 'object' and 'int' -- which is allowed, or would be if the bases were written in the opposite order. But still, something confuses the fragile logic of typeobject.c. (I'll leave this bit to scipy people to debug :-)
Well, I'm stumped on this. Note the method resolution order for the new
scalar array type (exactly as I would expect). Why doesn't the int32
type inherit its tp_free from the early types first?
a = zeros(10)
type(a[0]).mro()
[
Hi Travis, On Thu, Nov 24, 2005 at 10:17:43AM -0700, Travis E. Oliphant wrote:
Why doesn't the int32 type inherit its tp_free from the early types first?
In your case I suspect that the tp_free is inherited from the tp_base which is probably 'int'. I don't see how to "fix" typeobject.c, because I'm not sure that there is a solution that would do the right thing in all cases at this level. I would suggest that you just force the tp_alloc/tp_free that you want in your static types instead. That's what occurs for example if you build a similar inheritance hierarchy with classes defined in Python: these classes are then 'heap types', so they always get the generic tp_alloc/tp_free before PyType_Ready() has a chance to see them. Armin
Travis Oliphant wrote:
Bingo. Yes, definitely allocating new _types_ (an awful lot of them...) --- that's what the "array scalars" are: new types created in C.
are you allocating PyTypeObject structures dynamically? why are you creating an awful lot of new type objects to represent the contents of a homogenous array?
If they don't get properly collected then that would definitely have created the problem. It would seem this should be advertised when telling people to use PyObject_New for allocating new memory for an object.
PyObject_New creates a new instance of a given type; it doesn't, in itself, create a new type. at this point, your description doesn't make much sense. more information is definitely needed... </F>
Travis Oliphant
Bingo. Yes, definitely allocating new _types_ (an awful lot of them...) --- that's what the "array scalars" are: new types created in C.
Ah! And, er, why?
If they don't get properly collected then that would definitely have created the problem.
types do get collected -- but only after the cycle collector has run. If you can still reproduce the problem can you try again but calling 'gc.set_threshold(1)'?
It would seem this should be advertised when telling people to use PyObject_New for allocating new memory for an object.
Nevertheless, I think it would be good if pymalloc freed its arenas. I think the reasons it doesn't are because of worries that people might be called PyObject_Free without holding the GIL, but that's been verboten for several years now so we can probably just let them suffer. I think there's even a patch on SF to do this... Cheers, mwh -- The use of COBOL cripples the mind; its teaching should, therefore, be regarded as a criminal offence. -- Edsger W. Dijkstra, SIGPLAN Notices, Volume 17, Number 5
Hi Jim, You wrote:
(2) Is he allocating new _types_, which I think don't get properly collected.
(Off-topic) For reference, as far as I know new types are properly freed. There has been a number of bugs and lots of corner cases to fix, but I know of no remaining one. This assumes that the new types are heap types allocated in some official way -- either by Python code or by somehow calling type() from C. A bientot, Armin
participants (8)
-
"Martin v. Löwis" -
Armin Rigo -
Fredrik Lundh -
Michael Hudson -
Robert Kern -
Ronald Oussoren -
Travis E. Oliphant -
Travis Oliphant