[Tutor] memory management

[David Ascher]

On Sun, 18 Apr 1999, Christian Tismer wrote:

> (ick!) Is this really so? I thought I'd read this some time ago
> and never investigated further, since I had the experience
> that my WIndoze swapfile never shrunk after PythonWin had used
> and freed a large chunk of memory. When I shut down my session
> and wait a couple of seconds, I can see my swapfile shrunk again.
> Maybe this mislead me to make wrong assumptions.
> 
> Hmm. You are absolutely sure?
> Can it be an effect of memory fragmentation that it doesn't
> get released under circumstances?
>
> and watched my swapfile frow from small to 233 MB.
> Then I did a del on big. This took even more time than
> creating it, btw. Touching all the refcounts again seems
> to be quite a challenge for my harddisk. I have the strong
> impression that releasing objects should try to do this in
> reverse order.

Do what in reverse order?

> Without explicit measure, my guess is it takes
> 5 times as long to release this chunk.

My guess is that you're seeing OS-specific behavior.

> After a couple of seconds, my swapfile melted down to 192 MB.
> You are right. Some memory is freed. I'm just wondering about 
> the rest? After closing my PyWin session, the swap file quickly
> became as small as before. Can it be that I'm about to
> find a memory leak?

Maybe, but I'd be surprised.

A bit more about memory management (this is not really what I had in mind
for content for the tutor list, and most Python programmers should feel
free to ignore all of these issues, especially at first).

The general statement that Python releases memory when an object's
refcount drops to 0 is mostly true.

However, there are some special cases and caveats:

  - Python's memory management is somewhat object-type-specific -- some
    objects are allocated from buffers which are indeed not completely
    released when the objects aren't needed -- for example, frame objects
    (used internally) and integers are taken from preallocated buffers
    when possible, to speed up the common case of allocation and
    deallocation.  You might be seeing an effect of that with your range()
    test.  Details are in the code (see e.g. intobject.c and
    frameobject.c).  Interning of strings is another special-case behavior
    which can lead to apparent memory leask which aren't real memory
    leaks.

  - The vast majority of memory leaks in Python are due to the Python
    programmer forgetting to delete all the references to an object.  A
    special case of that is circular references.  E.g.:
  
         x = [1,2]
         y = {0:x}
         x[0] = y
  
    Now x refers to y and y refers to x -- so even if x and y are local
    variables in a function (which has a consequence that the named
    references would be removed when the function exits), then the
    "circular" or "mutual" references between the two will prevent Python
    from releasing the memory allocated to them.

    To solve this, one needs to 'break' the cycle, by e.g.:

         x[0] = None

  - When Python does free() a chunk of memory, it's up to the operating
    system (Linux, NT, Windows95, whatever) to decide how to deal with it.
    Some systems really free the memory right away, while others wait.

    It is very hard to find out exactly how much memory is being used by a
    Python program without knowing quite a bit about how the underlying
    system manages memory.  For example, I believe that Linux keeps in the
    "ps" table the high-watermark of the memory ever allocated by the
    program.  So, if you have a program which occasionally uses huge
    amounts of memory, it will have that amount of memory apparently
    allocated to that program.  In fact, Linux keeps track of what memory
    is in fact used by the program, and will allocate the freed memory to
    other programs as needed.

Cheers,

--david ascher

PS: I know that Christian at least knows some of the above -- I'm writing
    it more for didactic reasons.