python memory blow out

[jfj]

Stephen Thorne wrote:

> On Thu, 27 Jan 2005 09:08:59 +0800, Simon Wittber
> <simonwittber at gmail.com> wrote:
> 
>>According to the above post:
>>
>>>a) If the allocation is > 256 bytes, call the system malloc.
>>>b) If the allocation is < 256, use its own malloc implementation, which
>>>allocates memory in 256 kB chunks and never releases it.
>>
>>I imagine this means that large memory allocations are eventually
>>released back to the operating system. However, in my case, this
>>appears to be not happening.
> 
> 
> There was a recent patch posted to python-dev list which allows python
> to release memory back to the operating system once the 256kb chunk is
> no longer used.
> 

The policy is that the memory allocated for those things is as much as 
the maximum number of them where needed during the program.

This is "bad" in rare cases:

A program which
- at some point, while normally needs 10-20 integers, it peaks its 
requirements and allocates 10000000 integers.
- which does not terminate after that peak but keeps running for a long 
time without ever needing again many integers.

Such programs are rather rare. Moreover, the OS will happily swap out 
the unused int blocks after a while. A more pathetic situation would
be, in the above scenario to release all the 100000000 integers except 
from every 1000th. Assuming those ints left are accessed frequently,
the OS can't even swap out the pages!

But such programs, unless intentionally constructed are, **very** rare 
and it's supposed to be better to have a faster python in the general case.


Gerald.