-----Original Message----- From: twisted-python-bounces@twistedmatrix.com [mailto:twisted-python- bounces@twistedmatrix.com] On Behalf Of Maarten ter Huurne Sent: Monday, February 22, 2010 6:24 PM To: Twisted general discussion Subject: Re: [Twisted-Python] debugging a memory leak
On Tuesday 23 February 2010, Alec Matusis wrote:
When I start the process, both python object sizes and their counts rise proportionally to the numbers of reconnected clients, and then they stabilize after all clients have reconnected. At that moment, the "external" RSS process size is about 260MB. The "internal size" of all python objects reported by Heapy is about 150MB. After two days, the internal sizes/counts stay the same, but the external size grows to 1500MB.
Python object counts/total sizes are measured from the manhole. Is this sufficient to conclude that this is a C memory leak in one of
Hi Maarten, Your link http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete- a-large-object.htm seems to suggest that even though the interpreter does not release memory back to the OS, it can be re-used by the interpreter. If this was our problem, I'd expect the memory to be set by the highest usage, as opposed to it constantly leaking: in my case, the load is virtually constant, but the memory still leaks over time. The environment is Linux 2.6.24 x86-64, the extensions used are MySQLdb, pyCrypto (latest stable releases for both). the
external modules or in the Python interpreter itself?
In general, there are other reasons why heap size and RSS size do not match: 1. pages are empty but not returned to the OS 2. pages cannot be returned to the OS because they are not completely empty
It seems Python has different allocators for small and large objects: http://www.mail-archive.com/python-list@python.org/msg256116.html http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i- delete- a-large-object.htm
Assuming Python uses malloc for all its allocations (does it?), it is the malloc implementation that determines whether empty pages are returned to the OS. Under Linux with glibc (your system?), empty pages are returned, so there reason 1 does not apply.
Depending on the allocation behaviour of Python, the pages may not be empty though, so reason 2 is a likely suspect.
Python extensions written in C could also leak or fragment memory. Are you using any extensions that are not pure Python?
Bye, Maarten
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python