[ python-Bugs-1092502 ] Memory leak in socket.py on Mac OS X 10.3

SourceForge.net noreply at sourceforge.net
Sun Jan 2 00:01:06 CET 2005

Bugs item #1092502, was opened at 2004-12-28 21:09
Message generated for change (Comment added) made by bacchusrx
You can respond by visiting: 

Category: Python Library
Group: Platform-specific
Status: Open
Resolution: None
Priority: 5
Submitted By: bacchusrx (bacchusrx)
Assigned to: Nobody/Anonymous (nobody)
Summary: Memory leak in socket.py on Mac OS X 10.3

Initial Comment:
Some part of socket.py leaks memory on Mac OS X 10.3 (both with 
the python 2.3 that ships with the OS and with python 2.4).

I encountered the problem in John Goerzen's offlineimap. 
Transfers of messages over a certain size would cause the program 
to bail with malloc errors, eg

*** malloc: vm_allocate(size=5459968) failed (error code=3)
*** malloc[13730]: error: Can't allocate region

Inspecting the process as it runs shows that python's total memory
size grows wildly during such transfers.

The bug manifests in _fileobject.read() in socket.py. You can 
replicate the problem easily using the attached example with "nc -l 
-p 9330 < /dev/zero" running on some some remote host.

The way _fileobject.read() is written, socket.recv is called with the 
larger of the minimum rbuf size or whatever's left to be read. 
Whatever is received is then appended to a buffer which is joined 
and returned at the end of function.

It looks like each time through the loop, space for recv_size is 
allocated but not freed, so if the loop runs for enough iterations, 
python exhausts the memory available to it.

You can sidestep the condition if recv_size is small (like 
_fileobject.default_bufsize small).

I can't replicate this problem with python 2.3 on FreeBSD 4.9 or  
FreeBSD 5.2, nor on Mac OS X 10.3 if the logic from 
_fileobject.read() is re-written in Perl (for example).


>Comment By: bacchusrx (bacchusrx)
Date: 2005-01-01 18:01

Logged In: YES 

I've been able to replicate the problem reliably on both 10.3.5 and 
10.3.7. I've attached two more examples to demonstrate:

Try this: Do, "dd if=/dev/zero of=./data bs=1024 count=10240" and save 
server.pl wherever you put "data". Have three terminals open. In one, 
run "perl server.pl -s0.25". In another, run "top -ovsize" and in the third 
run "python example2.py". 

After about 100 iterations, python's vsize is +1GB (just about the value 
of cumulative_req in example2.py) and if left running will cause a 
malloc error at around 360 iterations with a vsize over 3.6GB (again, just 
about what cumulative_req reports). Mind you, we've only received 

server.pl differs from the netcat method in that it (defaults) to sending 
only 1492 bytes at a time (configurable with the -b switch) and sleeps for 
however many seconds specified with the -s switch. This guarantees 
enough iterations to raise the error each time around. When omittting 
the -s switch to server.pl, I don't get the error, but throughput is good 
enough that the loop in readFromSockUntil() only runs a few times.


Comment By: Bob Ippolito (etrepum)
Date: 2005-01-01 01:27

Logged In: YES 

I just played with a bit more.  If I catch the MemoryError and try again, 
most of the time it will work (sometimes on the second try).  These 
malloc faults seem to be some kind of temporary condition.


Comment By: Bob Ippolito (etrepum)
Date: 2005-01-01 01:18

Logged In: YES 

I can't reproduce this on either version of Python a 10.3.7 machine w/ 
1gb ram.  Python's total memory usage seems stable to me even if the 
read is in a while loop.

I can't see anything in sock_recv or _fileobject.read that will in any way 
leak memory.

With a really large buffer size (always >17mb, but it does vary with each 
run) it will get a memory error but the Python process doesn't grow 
beyond 50mb at the samples I looked at.  That's pretty much the amount 
of RAM I'd expect it to use.  

It is kind of surprising it doesn't want to allocate a buffer of that size, 
because I have the RAM for it.. but I don't think this is a bug.


You can respond by visiting: 

More information about the Python-bugs-list mailing list