[Python-Dev] Darwin's realloc(...) implementation never shrinks allocations

[Bob Ippolito]

Quite a few notable places in the Python sources expect realloc(...) to 
relinquish some memory if the requested size is smaller than the 
currently allocated size.  This is definitely not true on Darwin, and 
possibly other platforms.  I have tested this on OpenBSD and Linux, and 
the implementations on these platforms do appear to relinquish memory, 
but I didn't read the implementation.  I haven't been able to find any 
documentation that states that realloc should make this guarantee, but 
I figure Darwin does this as an "optimization" and because Darwin 
probably can't resize mmap'ed memory (at least it can't from Python, 
but this probably means it doesn't have this capability at all).

It is possible to "fix" this for Darwin, because you can ask the 
default malloc zone how big a particular allocation is, and how big an 
allocation of a given size will actually be (see: <malloc/malloc.h>).  
The obvious place to put this would be PyObject_Realloc, because this 
is at least called by _PyString_Resize (which will fix 
<http://python.org/sf/1092502>).

Should I write up a patch that "fixes" this?  I guess the best thing to 
do would be to determine whether the fix should be used at runtime, by 
allocating a meg or so, resizing it to 1 byte, and see if the size of 
the allocation changes.  If the size of the allocation does change, 
then the system realloc can be trusted to do what Python expects it to 
do, otherwise realloc should be done "cleanly" by allocating a new 
block (returning the original on failure, because it's good enough and 
some places in Python seem to expect that shrink will never fail), 
memcpy, free, return new block.

I wrote up a small hack that does this realloc indirection to CVS 
trunk, and it doesn't seem to cause any measurable difference in 
pystone performance.

Note that all versions of Darwin that I've looked at (6.x, 7.x, and 
8.0b1 corresponding to publicly available WWDC 2004 Tiger code) have 
this "issue", but it might go away by Mac OS X 10.4 or some later 
release.

This URL points to the sf bug and Darwin 7.7's realloc(...) 
implementation: 
http://bob.pythonmac.org/archives/2005/01/01/realloc-doesnt/

-bob