Limiting Python's Memory Use

[Vladimir Marangozov]

wpostma at my-deja.com wrote:
> 
> Here's a few questions for those knowledgeable in Python 1.5.2 source
> code internals. I could really use some help limiting the memory that
> Python holds on to.   Since I know Python is reference counted and not
> garbage-collected, it should be simple to avoid memory leaks, as long as
> there are no refcounting bugs. I believe I've found any refcounting bugs
> that would cause the system to release memory prematurely, but I may
> have orphans (leaked objects), or that Python itself consumes an
> unbounded amount of "working memory".
> 
> Here are my questions:
> 
> 1. Suppose I were to, one at a time over 24 hours, store rows into a
> disk database, using an auto-incrementing integer key to the database.
> Over 24 hours, the key value goes from 1 to over 200,000.  Would all
> 200,000 "integer objects" be created as needed, and then freed, or are
> the integer objects (200,000 of them) around for the duration of the
> python interpreter's lifetime?
> 

The memory for holding 200,000 integers will be reserved for the duration
of the interpreter's lifetime.

> 2. Suppose over 200,000 unique string constants were handled, are any of
> them free'd or are they kept around? I saw a cache for commonly used
> strings was created in the string object module, but I am unsure how
> this cache is limited, in size.

The cache is not limited in size. Some of the 200,000 unique strings
will be interned, the others won't. "abc" will be interned, "a space"
will not because it contains a space. The test for interning a string
constant is that of strspn() relative to "0..9A..Z_a..z".
This is done by the compiler.

> 
> 3. Is there any way to have Python report how much memory its modules
> are using?

Not yet. See a recent c.l.py thread "Memory monitoring under Python"
where I've addressed this concern.

> 
> 4. Can you tell Python to "suck itself in" somehow, or place limits, by
> recompiling python, on how many "spare" bits of memory it hangs on to?

Not yet. But an object memory allocator which can handle limits has
been proposed.

> 
> Some Details on What we're Doing:
> ...
> So far we are well in line, using only 16 mb RAM or less in all cases,
> except when the system makes heavy use of the python interpreter.
> ...
> information in memory.  However, a lot of different objects ( >200,000
> tuples, lists, dictionaries, etc) have been created, and hopefully
> disposed of during that time.  I suspect that Python isn't really
> releasing a lot of those disposed objects. Instead, it seems it keeps a
> cached list of freed objects, to re-use, saving the processing overhead
> of malloc'ing  and free'ing objects.

Right. Python uses simple free lists for some of its objects for
performance reasons: ints, small tuples, floats, frames. Simple free
lists is the worst scenario for people with memory limits, not interested
in performance, and with long running processes, because the free lists
reserve storage and never release that storage + they degenerate in time.
This is a killer for embedded systems with small memory. We're aware of
this and we're working on it.

Here are some hints:

1) use python -OO - this eliminates doc strings in bytecode
2) rewrite some object allocators so that they don't use free lists;
   this will slow down the interpreter, but will result in less memory
   consumption

   a) start with tuples and ints. You'll need to rewrite PyInt_FromLong
      int_dealloc, PyTuple_New, tuple_dealloc, so that they use malloc/free
      without free list caching.
   b) look at the allocators for lists & floats. You might want to change
      them as well.

3) for more hints, see http://www.abo.fi/~iporres/python/ presenting
   "Deeply embedded Python" -- a highly-stripped Python implementation

-- 
       Vladimir MARANGOZOV          | Vladimir.Marangozov at inrialpes.fr
http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252