[Python-bugs-list] [ python-Bugs-837231 ] Python 2.3 socketmodule breaks on FreeBSD 5.1

[SourceForge.net]

Bugs item #837231, was opened at 2003-11-06 16:03
Message generated for change (Comment added) made by mdornseif
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=837231&group_id=5470

Category: Extension Modules
Group: Python 2.3
Status: Open
Resolution: None
Priority: 7
Submitted By: Maximillian Dornseif (mdornseif)
Assigned to: Nobody/Anonymous (nobody)
Summary: Python 2.3 socketmodule breaks on FreeBSD 5.1

Initial Comment:
Every outbound socket created leaves a KQUEUE entry in the 
filetable which leads to a "out of ffiledescriport" problem dor 
long running programs or ones that do many connections.

I can reproduce this behaviour with Python 2.3 and 2.3.2 on 
FreeBSD 5.1. I have also tested 2.2.3 on FreeBSD 5.1 and 
2.3.2 on FreeBSD 4.9 which all were running fine.

Example:

import socket

def test():
    s = socket.socket()
    s.connect(('koeln.ccc.de', 80))
    s.send('GET / HTTP/1.1\r\nHost: koeln.ccc.de\r\n\r\n')
    # I know, I get only len(MTU)
    s.recv(30000)
    s.shutdown(2)
    s.close()
    del s

for i in range (500):
    test()


While running this I see about 500 times following lines in 
lsof:

python  82229   md    7u  KQUEUE 0xc7d76e00                  
count=0, state=0

Some more documentation on my tests can be found at 
http://blogs.23.nu/c0re/stories/1626/



 

----------------------------------------------------------------------

>Comment By: Maximillian Dornseif (mdornseif)
Date: 2003-11-07 08:26

Message:
Logged In: YES 
user_id=554460

Martin: KQUEUES are used by FreeBSD for event Notification like /
dev/poll on Linux. KQUEUES are exposed to the userland and can 
used for efficient non-blocking I/O - there is even a Python module 
for them. See http://people.freebsd.org/~jmg/kqueue.man.html 
for further enlightenment. To my understanding Python 2.3 itself 
ignores KQUEUES completely.

I SUSPECT FreeBSD 5 maps select and poll internally to code using 
KQUEUES.

Is this a Bug in FreeBSD? May be. But they will ask for a test 
program (in C) and I was not able to reproduce the problem in C 
or Perl. I also have found only one other mention of this problem 
and this was related to ntop which is doing all kind of funky stuff 
with your network; it also seemed they where not able to 
reproduce the problem.

So I think - as usual - the userland code has to work around bugs 
(Features?) in the kernel space. Especially since we (pythoneers) 
changed something between 2.2 and 2.3 which exposed the 
problem.

Anthony: I have only limited access to the FreeBSD 5.1. machine 
today, but it seems even socket().connect(); del results in the 
leak while socket(); del does not. 

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2003-11-06 21:56

Message:
Logged In: YES 
user_id=21627

What is a KQUEUE entry, and why do you think there is a bug
in Python (rather than in FreeBSD 5.1)?

----------------------------------------------------------------------

Comment By: Anthony Baxter (anthonybaxter)
Date: 2003-11-06 16:25

Message:
Logged In: YES 
user_id=29957

If you strip the test down further, which bits are necessary
for the leak? Is the full
socket()/connect()/send()/recv()/shutdown()/close() set
needed? What about just socket()/connect()/close() ? Or just
creating and del of the  sockets?


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=837231&group_id=5470