[Python-bugs-list] [ python-Bugs-837231 ] Python 2.3 socketmodule breaks on FreeBSD 5.1

[SourceForge.net]

Bugs item #837231, was opened at 2003-11-06 16:03
Message generated for change (Comment added) made by mdornseif
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=837231&group_id=5470

Category: Extension Modules
Group: Python 2.3
>Status: Closed
>Resolution: Invalid
Priority: 7
Submitted By: Maximillian Dornseif (mdornseif)
Assigned to: Nobody/Anonymous (nobody)
Summary: Python 2.3 socketmodule breaks on FreeBSD 5.1

Initial Comment:
Every outbound socket created leaves a KQUEUE entry in the 
filetable which leads to a "out of ffiledescriport" problem dor 
long running programs or ones that do many connections.

I can reproduce this behaviour with Python 2.3 and 2.3.2 on 
FreeBSD 5.1. I have also tested 2.2.3 on FreeBSD 5.1 and 
2.3.2 on FreeBSD 4.9 which all were running fine.

Example:

import socket

def test():
    s = socket.socket()
    s.connect(('koeln.ccc.de', 80))
    s.send('GET / HTTP/1.1\r\nHost: koeln.ccc.de\r\n\r\n')
    # I know, I get only len(MTU)
    s.recv(30000)
    s.shutdown(2)
    s.close()
    del s

for i in range (500):
    test()


While running this I see about 500 times following lines in 
lsof:

python  82229   md    7u  KQUEUE 0xc7d76e00                  
count=0, state=0

Some more documentation on my tests can be found at 
http://blogs.23.nu/c0re/stories/1626/



 

----------------------------------------------------------------------

>Comment By: Maximillian Dornseif (mdornseif)
Date: 2003-11-08 09:18

Message:
Logged In: YES 
user_id=554460

Ouch. Seems it really IS a FreeBSD Bug. While python2.2 
preformed yesterday as expected today it leaves KQUEUE entries, 
too. So now I have to try to reproduce this in C.

Sorry for wasting your time, I'm closing this bug.

(in case anybody is still interested: ktrace is available at http://
blogs.23.nu/c0re/stories/1626/#1632)



----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2003-11-07 20:01

Message:
Logged In: YES 
user_id=21627

Working around system bugs is not usual for Python, and I
would do so only if there was a clearly-documented bug of
the system, and a reliable way for testing for the bug's
presence. In the particular case of FreeBSD 5.1, I'm tempted
to say "tough luck", as the system is not considered ready
for production use by its developers.

That said: Please try to produce an
strace/truss/ktrace/whatever trace created when running the
Python program. There shouldn't be a need to have 500
repetitions - the leak of resources should occur in a single
repetition.

----------------------------------------------------------------------

Comment By: Maximillian Dornseif (mdornseif)
Date: 2003-11-07 10:43

Message:
Logged In: YES 
user_id=554460

It turs out, the same happens on scceptiong sockets. I have just 
looked into an xmlrpc server:

def submit_new_urls(uid, l):
    fn = os.path.join(destdir, "%s.newurls" % (uid))
    fd = open("%s.tmp" % fn, "w")
    fd.write("\n".join(l))
    fd.close()
    return "Ok"

server = SimpleXMLRPCServer(("example.com", 8000))
server.socket.setsockopt(socket.SOL_SOCKET, 
socket.SO_REUSEADDR, 0)
server.register_function(submit_new_urls)
server.serve_forever()    

same problem.

I will try to build a striped down test to demonstrate the problem.

----------------------------------------------------------------------

Comment By: Maximillian Dornseif (mdornseif)
Date: 2003-11-07 08:26

Message:
Logged In: YES 
user_id=554460

Martin: KQUEUES are used by FreeBSD for event Notification like /
dev/poll on Linux. KQUEUES are exposed to the userland and can 
used for efficient non-blocking I/O - there is even a Python module 
for them. See http://people.freebsd.org/~jmg/kqueue.man.html 
for further enlightenment. To my understanding Python 2.3 itself 
ignores KQUEUES completely.

I SUSPECT FreeBSD 5 maps select and poll internally to code using 
KQUEUES.

Is this a Bug in FreeBSD? May be. But they will ask for a test 
program (in C) and I was not able to reproduce the problem in C 
or Perl. I also have found only one other mention of this problem 
and this was related to ntop which is doing all kind of funky stuff 
with your network; it also seemed they where not able to 
reproduce the problem.

So I think - as usual - the userland code has to work around bugs 
(Features?) in the kernel space. Especially since we (pythoneers) 
changed something between 2.2 and 2.3 which exposed the 
problem.

Anthony: I have only limited access to the FreeBSD 5.1. machine 
today, but it seems even socket().connect(); del results in the 
leak while socket(); del does not. 

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2003-11-06 21:56

Message:
Logged In: YES 
user_id=21627

What is a KQUEUE entry, and why do you think there is a bug
in Python (rather than in FreeBSD 5.1)?

----------------------------------------------------------------------

Comment By: Anthony Baxter (anthonybaxter)
Date: 2003-11-06 16:25

Message:
Logged In: YES 
user_id=29957

If you strip the test down further, which bits are necessary
for the leak? Is the full
socket()/connect()/send()/recv()/shutdown()/close() set
needed? What about just socket()/connect()/close() ? Or just
creating and del of the  sockets?


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=837231&group_id=5470