[Pythonmac-SIG] Re: [Python-Dev] Darwin's realloc(...) implementation never shrinks allocations

[Bob Ippolito]

On Jan 4, 2005, at 5:56 AM, Jack Jansen wrote:

> On 3 Jan 2005, at 23:40, Bob Ippolito wrote:
>> Most people on Mac OS X have a lot of memory, and Mac OS X generally 
>> does a good job about swapping in and out without causing much of a 
>> problem, so I'm personally not very surprised that it could go 
>> unnoticed this long.
>
> *Except* when you're low on free disk space. 10.2 and before were 
> really bad with this, usually hanging the machine, 10.3 is better but 
> it's still pretty bad when compared to other unixen. It probably has 
> something to do with the way OSX overcommits memory and swapspace, for 
> which it apparently uses a different algorithm than FreeBSD or Linux.
>
> I wouldn't be surprised if the bittorrent problem report in this 
> thread was due to being low on diskspace. And that could also be true 
> for the original error report that sparked this discussion.

I was able to trigger this bug with a considerable amount of free disk 
space using a laptop that has 1GB of RAM, although I did have to 
increase the buffer size from the given example quite a bit to get it 
to fail.  After all, a 32-bit process can't have more than 4 GB of 
addressable memory.  I am pretty sure that OS X is never supposed to 
overcommit memory.  The disk thrashing probably has a lot to do with 
the fact that Mac OS X will grow and shrink its swap based on demand, 
rather than having a fixed size swap partition as is common on other 
unixen.  I've never seen the problem myself, though.

 From what I remember about Linux, its malloc implementation merely 
increases the address space of a process.  The actual allocation will 
happen when you try and access the memory, and if it's overcommitted 
things will fail in a bad way.

-bob