[Patches] [ python-Patches-1123430 ] Python memory allocator: Free memory
SourceForge.net
noreply at sourceforge.net
Fri Jun 30 08:17:06 CEST 2006
Patches item #1123430, was opened at 2005-02-15 16:27
Message generated for change (Comment added) made by tim_one
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1123430&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Core (C code)
Group: Python 2.5
Status: Closed
Resolution: Accepted
Priority: 5
Submitted By: Evan Jones (vulturex)
Assigned to: Tim Peters (tim_one)
Summary: Python memory allocator: Free memory
Initial Comment:
This is the second version of my Python memory allocator patch.
The first version was discussed on the python-dev mailing list
here:
http://mail.python.org/pipermail/python-dev/2005-January/
051255.html
This patch enables Python to actually return memory to the
system. The current version's memory usage will only grow. This
version maintains the same backwards compatability guarantees
as the previous version: Calling PyObject_Free with a pointer that
was returned by malloc() while NOT holding the GIL will work, and
will not corrupt the state of the memory allocator.
The patch modifies obmalloc.c. If it is accepted, other
modifications to that file are required. In particular, I have not yet
updated the WITH_MEMORY_LIMITS implementation, nor have I
looked closely at the PYMALLOC_DEBUG code to see what changes
(if any) are required.
----------------------------------------------------------------------
>Comment By: Tim Peters (tim_one)
Date: 2006-06-30 02:17
Message:
Logged In: YES
user_id=31435
As the NEWS entry says,
"""
Patch #1123430: Python's small-object allocator now returns
an arena to the system ``free()`` when all memory within an
arena becomes unused again. Prior to Python 2.5, arenas
(256KB chunks of memory) were never freed. Some
applications will see a drop in virtual memory size now,
especially long-running applications that, from time to
time, temporarily use a large number of small objects. Note
that when Python returns an arena to the platform C's
``free()``, there's no guarantee that the platform C library
will in turn return that memory to the operating system.
The effect of the patch is to stop making that impossible,
and in tests it appears to be effective at least on
Microsoft C and gcc-based systems.
"""
An instrumented debug build of current trunk showed that
Python eventually returned 472 of 544 allocated arenas to
the platform free() in your program, leaving 72 still
allocated when Python shut down. The latter is due to
fragmentation, and there will never be "a cure" for that
since CPython guarantees never to move objects. The arena
highwater mark was 240 arenas, which is what Python would
have held on to forever before the patch.
A large part of the relatively disappointing result we see
here is due to that the tuple implementation maintains its
own free lists too: it never returns a few thousand of the
tuple objects to obmalloc before Python shuts down, and that
in turn keeps all the arenas from which those tuple objects
were originally obtained alive until Python shuts down.
There's nothing obmalloc or gcmodule can do about that, and
the tuple free lists are _likely_ an important optimization
in real applications.
----------------------------------------------------------------------
Comment By: Rene Dudfield (illume)
Date: 2006-06-30 02:02
Message:
Logged In: YES
user_id=2042
I have done some more tests... and it seems that
dictionaries do not release as much memory as lists do.
Here is a modification of the last example posted.
If you only let fill2() run almost all the memory is freed.
fill2() uses lists. However if you let the others which
use dicts run not all of the memory is freed. The processes
are still 38MB when the data is del'ed.
It is still much better than python2.4, however I think
something fishy must be going on with dicts.
<pre>
import random
import os
def fill_this_one_doesnot_free_much_at_all(map):
random.seed(0)
for i in xrange(300000):
k1 = (random.randrange(100),
random.randrange(100),
random.randrange(100))
k2 = random.randrange(100)
if k1 in map:
d = map[k1]
else:
d = dict()
d[k2] = d.get(k2, 0) + 1
map[k1] = d
def fill(map):
random.seed(0)
for i in xrange(3000000):
map[i] = "asdf"
class Obj:
def __init__( self ):
self.dumb = "hello"
def fill2(map):
a = []
for i in xrange(300000):
o = Obj()
a.append(o)
return a
if __name__ == "__main__":
import time
import gc
os.system('ps v | grep memtest')
d = dict()
a = fill2(d)
#a2 = fill(d)
a2 = fill_this_one_doesnot_free_much_at_all(d)
print "-"* 50
print "\n" * 3
os.system('ps v | grep memtest')
del d
del a
gc.collect()
time.sleep(2)
for x in xrange(100000):
pass
print "-"* 50
print "\n" * 3
os.system('ps v | grep memtest')
</pre>
----------------------------------------------------------------------
Comment By: Rene Dudfield (illume)
Date: 2006-06-30 00:06
Message:
Logged In: YES
user_id=2042
Note, that this patch doesn't fix all memory leaks of this
type. For example, this code below doesn't release all the
memory after it is run. It starts at about 3MB, goes up to
about 56MB and then only drops to 50MB.
AFAIK restarting python processes is still needed to reduce
memory usage for certain types of processes.
<pre>
import random
import os
def fill(map):
random.seed(0)
for i in xrange(300000):
k1 = (random.randrange(100),
random.randrange(100),
random.randrange(100))
k2 = random.randrange(100)
if k1 in map:
d = map[k1]
else:
d = dict()
d[k2] = d.get(k2, 0) + 1
map[k1] = d
if __name__ == "__main__":
os.system('ps v')
d = dict()
fill(d)
print "-"* 50
print "\n" * 3
os.system('ps v')
del d
import gc
gc.collect()
print "-"* 50
print "\n" * 3
os.system('ps v')
</pre>
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2006-03-15 20:17
Message:
Logged In: YES
user_id=31435
The tim-obmalloc branch was merged to the trunk (for Python
2.5a1) in revision 43059. Thank you again for your hard
work and patience, Evan!
----------------------------------------------------------------------
Comment By: Evan Jones (vulturex)
Date: 2006-02-23 09:29
Message:
Logged In: YES
user_id=539295
Great news! If you need any assistance, I would be more than
happy to help.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2006-02-22 20:25
Message:
Logged In: YES
user_id=31435
The patch here is out of date, but that's OK. I created
branch tim-obmalloc, with a working version of the patch,
extensively reformatted to Python's C style, and with some
minor changes to squash compiler warnings. I plan to finish
this during PyCon.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2005-09-05 18:43
Message:
Logged In: YES
user_id=31435
Assigned to me.
----------------------------------------------------------------------
Comment By: Evan Jones (vulturex)
Date: 2005-05-10 00:31
Message:
Logged In: YES
user_id=539295
Whoops! I uploaded a "fixed" version a while ago, but I guess I didn't
update the comments. The patch currently attached to this is the most up-
to-date version. Sorry about that.
----------------------------------------------------------------------
Comment By: Evan Jones (vulturex)
Date: 2005-02-19 09:07
Message:
Logged In: YES
user_id=539295
As per the discussion on python-dev, I've removed the concurrency hack.
The routines in obmalloc.c now *must* be called while holding the GIL,
even if the pointer was allocated with malloc(). I also finally fixed the
PYMALLOC_DEBUG routines, so I believe this patch is now "complete."
----------------------------------------------------------------------
Comment By: Evan Jones (vulturex)
Date: 2005-02-18 17:08
Message:
Logged In: YES
user_id=539295
Please ignore this patch for the moment: I'm in the process of making
some fixes.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1123430&group_id=5470
More information about the Patches
mailing list