[ python-Bugs-1092502 ] Memory leak in socket.py on Mac OS X 10.3

[SourceForge.net]

Bugs item #1092502, was opened at 2004-12-28 21:09
Message generated for change (Comment added) made by bacchusrx
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1092502&group_id=5470

Category: Python Library
Group: Platform-specific
Status: Open
Resolution: None
Priority: 5
Submitted By: bacchusrx (bacchusrx)
Assigned to: Nobody/Anonymous (nobody)
Summary: Memory leak in socket.py on Mac OS X 10.3

Initial Comment:
Some part of socket.py leaks memory on Mac OS X 10.3 (both with 
the python 2.3 that ships with the OS and with python 2.4).

I encountered the problem in John Goerzen's offlineimap. 
Transfers of messages over a certain size would cause the program 
to bail with malloc errors, eg

*** malloc: vm_allocate(size=5459968) failed (error code=3)
*** malloc[13730]: error: Can't allocate region

Inspecting the process as it runs shows that python's total memory
size grows wildly during such transfers.

The bug manifests in _fileobject.read() in socket.py. You can 
replicate the problem easily using the attached example with "nc -l 
-p 9330 < /dev/zero" running on some some remote host.

The way _fileobject.read() is written, socket.recv is called with the 
larger of the minimum rbuf size or whatever's left to be read. 
Whatever is received is then appended to a buffer which is joined 
and returned at the end of function.

It looks like each time through the loop, space for recv_size is 
allocated but not freed, so if the loop runs for enough iterations, 
python exhausts the memory available to it.

You can sidestep the condition if recv_size is small (like 
_fileobject.default_bufsize small).

I can't replicate this problem with python 2.3 on FreeBSD 4.9 or  
FreeBSD 5.2, nor on Mac OS X 10.3 if the logic from 
_fileobject.read() is re-written in Perl (for example).

----------------------------------------------------------------------

>Comment By: bacchusrx (bacchusrx)
Date: 2005-01-01 18:01

Message:
Logged In: YES 
user_id=646321

I've been able to replicate the problem reliably on both 10.3.5 and 
10.3.7. I've attached two more examples to demonstrate:

Try this: Do, "dd if=/dev/zero of=./data bs=1024 count=10240" and save 
server.pl wherever you put "data". Have three terminals open. In one, 
run "perl server.pl -s0.25". In another, run "top -ovsize" and in the third 
run "python example2.py". 

After about 100 iterations, python's vsize is +1GB (just about the value 
of cumulative_req in example2.py) and if left running will cause a 
malloc error at around 360 iterations with a vsize over 3.6GB (again, just 
about what cumulative_req reports). Mind you, we've only received 
~512kbytes.

server.pl differs from the netcat method in that it (defaults) to sending 
only 1492 bytes at a time (configurable with the -b switch) and sleeps for 
however many seconds specified with the -s switch. This guarantees 
enough iterations to raise the error each time around. When omittting 
the -s switch to server.pl, I don't get the error, but throughput is good 
enough that the loop in readFromSockUntil() only runs a few times.

----------------------------------------------------------------------

Comment By: Bob Ippolito (etrepum)
Date: 2005-01-01 01:27

Message:
Logged In: YES 
user_id=139309

I just played with a bit more.  If I catch the MemoryError and try again, 
most of the time it will work (sometimes on the second try).  These 
malloc faults seem to be some kind of temporary condition.

----------------------------------------------------------------------

Comment By: Bob Ippolito (etrepum)
Date: 2005-01-01 01:18

Message:
Logged In: YES 
user_id=139309

I can't reproduce this on either version of Python a 10.3.7 machine w/ 
1gb ram.  Python's total memory usage seems stable to me even if the 
read is in a while loop.

I can't see anything in sock_recv or _fileobject.read that will in any way 
leak memory.

With a really large buffer size (always >17mb, but it does vary with each 
run) it will get a memory error but the Python process doesn't grow 
beyond 50mb at the samples I looked at.  That's pretty much the amount 
of RAM I'd expect it to use.  

It is kind of surprising it doesn't want to allocate a buffer of that size, 
because I have the RAM for it.. but I don't think this is a bug.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1092502&group_id=5470