[Patches] [ python-Patches-1123430 ] Python memory allocator: Free memory

[SourceForge.net]

Patches item #1123430, was opened at 2005-02-15 21:27
Message generated for change (Comment added) made by illume
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1123430&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Core (C code)
Group: Python 2.5
Status: Closed
Resolution: Accepted
Priority: 5
Submitted By: Evan Jones (vulturex)
Assigned to: Tim Peters (tim_one)
Summary: Python memory allocator: Free memory

Initial Comment:
This is the second version of my Python memory allocator patch. 
The first version was discussed on the python-dev mailing list 
here:

http://mail.python.org/pipermail/python-dev/2005-January/
051255.html

This patch enables Python to actually return memory to the 
system. The current version's memory usage will only grow. This 
version maintains the same backwards compatability guarantees 
as the previous version: Calling PyObject_Free with a pointer that 
was returned by malloc() while NOT holding the GIL will work, and 
will not corrupt the state of the memory allocator.

The patch modifies obmalloc.c. If it is accepted, other 
modifications to that file are required. In particular, I have not yet 
updated the WITH_MEMORY_LIMITS implementation, nor have I 
looked closely at the PYMALLOC_DEBUG code to see what changes 
(if any) are required.


----------------------------------------------------------------------

Comment By: Rene Dudfield (illume)
Date: 2006-06-30 04:06

Message:
Logged In: YES 
user_id=2042


Note, that this patch doesn't fix all memory leaks of this
type.  For example, this code below doesn't release all the
memory after it is run.  It starts at about 3MB, goes up to
about 56MB and then only drops to 50MB.

AFAIK restarting python processes is still needed to reduce
memory usage for certain types of processes.


<pre>

import random
import os

def fill(map):
    random.seed(0)
    for i in xrange(300000):
        k1 = (random.randrange(100),
              random.randrange(100),
              random.randrange(100))
        k2 = random.randrange(100)
        if k1 in map:
            d = map[k1]
        else:
            d = dict()
        d[k2] = d.get(k2, 0) + 1
        map[k1] = d

if __name__ == "__main__":
    os.system('ps v')
    d = dict()
    fill(d)
    print "-"* 50
    print "\n" * 3
    os.system('ps v')
    del d

    import gc
    gc.collect()

    print "-"* 50
    print "\n" * 3
    os.system('ps v')
</pre>

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2006-03-16 01:17

Message:
Logged In: YES 
user_id=31435

The tim-obmalloc branch was merged to the trunk (for Python
2.5a1) in revision 43059.  Thank you again for your hard
work and patience, Evan!

----------------------------------------------------------------------

Comment By: Evan Jones (vulturex)
Date: 2006-02-23 14:29

Message:
Logged In: YES 
user_id=539295

Great news! If you need any assistance, I would be more than
happy to help.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2006-02-23 01:25

Message:
Logged In: YES 
user_id=31435

The patch here is out of date, but that's OK.  I created
branch tim-obmalloc, with a working version of the patch,
extensively reformatted to Python's C style, and with some
minor changes to squash compiler warnings.  I plan to finish
this during PyCon.

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2005-09-05 22:43

Message:
Logged In: YES 
user_id=31435

Assigned to me.

----------------------------------------------------------------------

Comment By: Evan Jones (vulturex)
Date: 2005-05-10 04:31

Message:
Logged In: YES 
user_id=539295

Whoops! I uploaded a "fixed" version a while ago, but I guess I didn't 
update the comments. The patch currently attached to this is the most up-
to-date version. Sorry about that.

----------------------------------------------------------------------

Comment By: Evan Jones (vulturex)
Date: 2005-02-19 14:07

Message:
Logged In: YES 
user_id=539295

As per the discussion on python-dev, I've removed the concurrency hack. 
The routines in obmalloc.c now *must* be called while holding the GIL, 
even if the pointer was allocated with malloc(). I also finally fixed the 
PYMALLOC_DEBUG routines, so I believe this patch is now "complete."

----------------------------------------------------------------------

Comment By: Evan Jones (vulturex)
Date: 2005-02-18 22:08

Message:
Logged In: YES 
user_id=539295

Please ignore this patch for the moment: I'm in the process of making 
some fixes.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1123430&group_id=5470