[Python-Dev] Changing pymalloc behaviour for long running processes

[Dan Sugalski]

At 12:14 PM -0400 10/19/04, Tim Peters wrote:
>[Evan Jones]
>  > The short version of the problem is that obmalloc.c never frees memory.
>
>True.  That's one major problem for some apps.  Another major problem
>for some apps is due to unbounded internal free lists outside of
>obmalloc.  Another is that the platform OS+libc may not shrink VM at
>times even when memory is returned to the system free().

FWIW, at this point nearly all OSes have a means of allocating memory 
from the system that can then later be returned to the system. 
(malloc and free tend *not* to do this) Even on Unix platforms you 
can play the "mmap a file with no filename" game to get returnable 
chunks. Unfortunately there's often a limit to the number of these 
chunks you can get from the OS so it's not safe to unconditionally 
replace malloc and free. (The performance impact isn't worth it 
either)

If someone's going to do this, I'd suggest the place to start is 
adding separate allocate and free API entry points for returnable 
chunks and put in some criteria for getting memory from them 
(allocation size, particular spots that allocate, or whatever) and 
see where you go from there.

I'll point out that, from experience, this can be a non-trivial 
thing, and with a non-moving GC system you'll probably find that 
there are relatively few places where there's a win from it. It does, 
though, tend to flush out dangling pointers. (Whether this is good or 
bad is a separate issue, of course ;)
-- 
				Dan

--------------------------------------it's like this-------------------
Dan Sugalski                          even samurai
dan at sidhe.org                         have teddy bears and even
                                       teddy bears get drunk