[Twisted-Python] Buildbot history
Hey all, Due to some unfortunate behavior which began after I upgraded our build master to 0.7.8, I've just removed all of the history from our buildbot. This means all old links to build results are broken now. Hopefully this won't cause any major problems. Jean-Paul
As I wrote in post http://www.mail-archive.com/twisted-web@twistedmatrix.com/msg01701.html I'm still observing this socket error 22 problem but was not able to dig into it any further until now. I found the following post from the python mailing list, which seems related to my problem: http://mail.python.org/pipermail/python-bugs-list/2006-August/034667.html Attchached to this post is a small script, which I extended somehow and tried out on various platforms: import socket socket.setdefaulttimeout(30) s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) #nothing is listening on port 9999 try: s.connect(('127.0.0.1', 9999)) except socket.error, msg: print msg print s.getsockname() Win32 XP & VISTA: (10061, 'Connection refused') ('0.0.0.0', 2071) << socket# seems to be random Linux Fedora 9: (111, 'Connection refused') ('0.0.0.0', 47521) << socket# counting up when script run again Mac OSX 10.5: (61, 'Connection refused') ('127.0.0.1', 49169) << socket# counting up when script run again FreeBSD 7.0: (61, 'Connection refused') ('127.0.0.1', 0) << socket# remains 0 when script run again OpenBSD 4.3: (61, 'Connection refused') Traceback (most recent call last): File "socket22error.py", line 8, in <module> print s.getsockname() File "<string>", line 1, in getsockname socket.error: (22, 'Invalid argument') All the OS's seem to agree more or less how this particular situation should be handled except OpenBSD. I agree that this is an Python/OS related question, but what happens with the reactor when such an error is thrown? Is it harmless or does the system loose resources, wait for timeouts? Is a try/except with a normalized answer in the embracing function in twisted/internet/tcp.py a possible cure? I ask because I experience an unexplainable 'hickup' behavior where the odd request which usually triggers an instantaneous response takes up to 30secs until a response is seen on the client in a local and clean net enviroment. Thxs, Werner PS: - Yes, I know trial. - Yes, I am able to set up a build slave for OpenBSD and in fact did so. - Yes, I requested the creds from exarkun@divmod.com for connecting this buildslave on 03.July 2008 17:37 but got no answer - Yes, I'm willing to recreate the bot which was in the meantime used in another project - No, I don't know if there exists an automated test case for the problem given above
On Wed, 08 Oct 2008 18:57:51 +0200, Werner Thie <wthie@thiengineering.ch> wrote:
As I wrote in post http://www.mail-archive.com/twisted- web@twistedmatrix.com/msg01701.html I'm still observing this socket error 22 problem but was not able to dig into it any further until now.
I found the following post from the python mailing list, which seems related to my problem:
http://mail.python.org/pipermail/python-bugs-list/2006-August/034667.html
Attchached to this post is a small script, which I extended somehow and tried out on various platforms:
[snip]
It looks like these are slightly different cases. The traceback from Twisted Web comes from using getsockname on a connection the server accepted. The script you attached calls getsockname on a socket which has failed to make an outgoing connection. Perhaps they are demonstrating the same feature/behavior/bug of OpenBSD, but not necessarily. I vaguely recall trying to investigate the Twisted Web case when you initially raised it, but I can't remember if I learned anything (I think I recall that I was not able to reproduce it).
[snip]
I agree that this is an Python/OS related question, but what happens with the reactor when such an error is thrown?
Is it harmless or does the system loose resources, wait for timeouts?
When a protocol's dataReceived method raises an exception, the connection is closed. It seems that this exception is getting all the way from the request handling code up to beyond the HTTP protocol implementation's dataReceived method, so that should be what happens for this case. That means it's probably harmless, but since I don't actually understand the circumstances under which it happens, I can't say for sure.
Is a try/except with a normalized answer in the embracing function in twisted/internet/tcp.py a possible cure?
You mean in the implementation of getHost()? Maybe, maybe not. I'd like to understand what OpenBSD thinks it is telling us when it returns EINVAL here.
I ask because I experience an unexplainable 'hickup' behavior where the odd request which usually triggers an instantaneous response takes up to 30secs until a response is seen on the client in a local and clean net enviroment.
It could be that this exception is preventing a request from being processed, or there could be some other problem, I don't know. Jean-Paul
Is a try/except with a normalized answer in the embracing function in twisted/internet/tcp.py a possible cure?
You mean in the implementation of getHost()? Maybe, maybe not. I'd like to understand what OpenBSD thinks it is telling us when it returns EINVAL here.
According to several 10 Gig of logfiles the error occurs in all kind of situations be it POST or GET and always with the same traceback. What I also see is that errors are more frequent when loads are higher, which leads me to the conclusion that this might be a resource dependency specially the fd count per process which we currently have at 8192 as I already suspected in our previous discussion end of April. I agree with you that my test script is not exactly the same situation and the test you sent (just for the sake of completeness) me was inconclusive: ---- Paul's code ------------------------------------------------------ I wonder if OpenBSD gives back EINVAL for getsockname(2) when the peer has done a shutdown of the socket. Can you tell me what this program does? from socket import socket port = socket() port.bind(('', 0)) port.listen(1) client = socket() client.setblocking(False) client.connect_ex(port.getsockname()) server, clientAddr = port.accept() print 'Before close', server.getsockname() client.close() print 'After close', server.getsockname() If it finishes without error, try it with a time.sleep(5) before the final line. ---- end of Paul's code ------------------------------------------------ After a lot of googling there seems to be a possible onset of the problem dating back to 2006 shortly before the 4.3 release. There are other users reporting similar 'hickups' I experience on SSH and FTP, but none of those reports actually lead to a fix probably because everybody is used to a 'slow login', a typein hickup or a failed ftp. One might speculate that this could be a hardware related error but us having several servers with very different HW under similar loads and observing those errors it might be a race condition in the kernel driver. Also odd enough that the man pages for OpenBSD are the only one among the **xes who do not mention EINVAL as a possible return value but in fact do return it. I'll try to file a bug with OpenBSD but doubt that it will get much attention being so impossible to trigger it. Does somebody which is more fit on TCP/IP have an additional idea how to trigger such a behavior as described in my previous posts. Traceback below for completeness Thxs, Werner Running on twisted 2.5 OpenBSD I see the following traceback in the logs every twenty minutes with up to 200 athena/LivePages connected: Traceback (most recent call last): --- <exception caught here> --- File "/usr/local/lib/python2.5/site-packages/twisted/internet/selectreactor.py", line 148, in _doReadOrWrite why = getattr(selectable, method)() File "/usr/local/lib/python2.5/site-packages/twisted/internet/tcp.py", line 362, in doRead return self.protocol.dataReceived(data) File "/usr/local/lib/python2.5/site-packages/twisted/protocols/basic.py", line 233, in dataReceived why = self.lineReceived(line) File "/usr/local/lib/python2.5/site-packages/twisted/web/http.py", line 1069, in lineReceived self.allContentReceived() File "/usr/local/lib/python2.5/site-packages/twisted/web/http.py", line 1110, in allContentReceived req.requestReceived(command, path, version) File "/usr/local/lib/python2.5/site-packages/twisted/web/http.py", line 603, in requestReceived self.host = self.channel.transport.getHost() File "/usr/local/lib/python2.5/site-packages/twisted/internet/tcp.py", line 672, in getHost return address.IPv4Address('TCP', *(self.socket.getsockname() + ('INET',))) File "<string>", line 1, in getsockname socket.error: (22, 'Invalid argument')
participants (2)
-
Jean-Paul Calderone -
Werner Thie