Limiting Python's Memory Use

[Vladimir Marangozov]

Johannes Stezenbach wrote:
> 
> [me, saving keystrokes]
> >The memory for holding 200,000 integers will be reserved for the duration
> >of the interpreter's lifetime.
> 
> I think this needs clarification:
> 
> When you count a variable from 1 up to 200,000 each integer
> object gets freed (added to the freelist) before (well, shortly
> after in realitiy) the next one is created (from the freelist),
> so you need only one (or two or three for temporary storage)
> integer objects, not 200,000.
> 
> That's much different from creating all 200,000 at once, like in
> range(200000).

Right. We both lied loud. Space for the ints' free list is allocated in
chunks of 1k to save malloc overhead. The reserved mem size is the one
which can hold the *peak* number of different ints allocated simultaneously
any point in time.

> 
> Free lists mean that not all memory that your script has used
> will be freed. It means *not* that memory consumtion will grow
> limitless because all garbage is kept around, just in case
> someone could use it.
> Some wasted memory usually won't be a problem for most (short or
> long running) scripts.

Clarification, to emphasize one point: wasted memory *is* a big problem
for any system without paging (mem sharing) or without secondary storage.

Second, what's important is not how much memory is used + "wasted" (a.k.a.
reserved and unused, due to fragmentation and/or suboptimal management)
at some point of the execution. Few people care about the current memory
footprint. What's important is the *peak* memory consumption. If we
request only 1 byte more than the peak amount the system can allocate
for us, we're dead. And *this* is usually a real problem for long
running proceses, even on modern systems with gobs of memory and
sophisticated memory managers.

-- 
       Vladimir MARANGOZOV          | Vladimir.Marangozov at inrialpes.fr
http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252