[Twisted-Python] Plain windows sockets and twisted
Hi, I am using twisted in a cross-platform manner. I have written both the client and server. The client is twisted and the server uses python threads. I am using the select reactor and plain old TCP connections. I encode my messages to JSON to send them. On Windows I get tons of un-clean connectionlost events, on Mac everything works perfectly. I am using the same data set in each case. If I use the threaded server on windows and the twisted client on mac then I still get tons of errors. 99% of these times these errors happen after the last dataReceived when the socket should be closed cleanly. My threaded code is: try: (clientsocket, address) = serversocket.accept() except Exception, e: print "Exception ", e, clientsocket try: p = processrequest() p.set_socket(clientsocket) p.start() and in processrequest self.sock.send_all(my_json_message) self.sock.close() Is this the correct behavior for using plain sockets with twisted? Any reason why this would be perfectly fine on OSX and not Windows? Sorry for the long email. Thank you very very much, Dan
On Jan 17, 2010, at 6:25 PM, Daniel Griffin wrote:
I am using twisted in a cross-platform manner. I have written both the client and server. The client is twisted and the server uses python threads. I am using the select reactor and plain old TCP connections. I encode my messages to JSON to send them.
On Windows I get tons of un-clean connectionlost events, on Mac everything works perfectly. I am using the same data set in each case.
You should use Twisted for your server and get rid of threads, or you should very carefully read <http://itamarst.org/writings/win32sockets.html> and understand the differences between what windows calls "sockets" and what the rest of the world does.
You'll need to give some more detail. What errors are you seeing? Please post a complete traceback, along with enough code to actually trigger the error. (<http://sscce.org/> is a good resource explaining desirable attributes of such an example.) What version of Twisted are you using? What version of Python? What is the protocol? Are the errors occurring in your threaded server or your twisted client? The quick answer to your question is: no, this behavior is not expected. Sockets are sockets. Twisted should work fine talking to sockets in threads, sockets in other programming languages, other operating systems, etc. Of course, that's just what's *supposed* to happen: there's a bug here, and with the information I have now, I can't say whether it's in your threaded code, your Twisted code, Twisted, or even Python. Good luck?
Thank you for the response. I am using twisted 9.0 and tried 8.2. The python version is 2.6. When I print the reason sent to connection lost I get "[Failure instance: Traceback (failure with no frames): <class 't wisted.internet.error.ConnectionLost' Connection to the other side was lost in a non-clean fashion.". I am using IP4 TCP sockets. I realize python threads are defective but in this case they are easier to deal with. I will create a chunk of sample code and send it to the list. Thanks, Dan On Sun, Jan 17, 2010 at 6:52 PM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
Here are two tiny apps. They run perfectly on OSX and error like this on windows: Log conenction [Failure instance: Traceback (failure with no frames): <class 't wisted.internet.error.ConnectionLost' Connection to the other side was lost in a non-clean fashion. ] Log conenction [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost' Connection to the other side was lost in a non-clean fashion. .... Start server.py then run tclient.py. Dan On Sun, Jan 17, 2010 at 8:55 PM, Daniel Griffin <dgriff1@gmail.com> wrote:
On Jan 17, 2010, at 10:45 PM, Daniel Griffin wrote:
Ah. This is very simple, and luckily has nothing to do with threads. The simplest explanation is that these errors are showing up because you are printing them out :). If you stop printing them out, they won't show up. More specifically: You generally shouldn't worry about "clean" vs. "non-clean" connection shutdown. There are lots of things that can cause "clean" shutdowns to still lose data, and there are many cases where a "non-clean" shutdown is actually fine. Your application protocol should be framing messages such that you can tell which ones have been responded to. That said, what's going on here is pretty simple. A "clean" shutdown is, broadly speaking, when both ends of the TCP connection agree on where the stream of bytes begins and ends. I send you some data, you read all of it, I shut down the connection, you shut down the connection. If I send you some data, and you *don't* read all of it, then I shut the connection down, the result will be a "non-clean" shutdown, because there is still data outstanding when the connection dies. In this case, you are calling recv(5) in your server, but your client is writing "RESPOND" to its transport. As you can see,
len("RESPOND") 7
"RESPOND" is more than 5 bytes. If you were using Twisted for your server, this wouldn't happen because Twisted always reads all the bytes that are available and delivers them to dataReceived.
Awesome, in my actual code I already implemented something to see if I was "done" with the socket conversation. I was just getting really worried about the "non-clean" messages. Your explanation makes perfect sense. Thanks for the help. Dan On Sun, Jan 17, 2010 at 11:55 PM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
On Jan 17, 2010, at 6:25 PM, Daniel Griffin wrote:
I am using twisted in a cross-platform manner. I have written both the client and server. The client is twisted and the server uses python threads. I am using the select reactor and plain old TCP connections. I encode my messages to JSON to send them.
On Windows I get tons of un-clean connectionlost events, on Mac everything works perfectly. I am using the same data set in each case.
You should use Twisted for your server and get rid of threads, or you should very carefully read <http://itamarst.org/writings/win32sockets.html> and understand the differences between what windows calls "sockets" and what the rest of the world does.
You'll need to give some more detail. What errors are you seeing? Please post a complete traceback, along with enough code to actually trigger the error. (<http://sscce.org/> is a good resource explaining desirable attributes of such an example.) What version of Twisted are you using? What version of Python? What is the protocol? Are the errors occurring in your threaded server or your twisted client? The quick answer to your question is: no, this behavior is not expected. Sockets are sockets. Twisted should work fine talking to sockets in threads, sockets in other programming languages, other operating systems, etc. Of course, that's just what's *supposed* to happen: there's a bug here, and with the information I have now, I can't say whether it's in your threaded code, your Twisted code, Twisted, or even Python. Good luck?
Thank you for the response. I am using twisted 9.0 and tried 8.2. The python version is 2.6. When I print the reason sent to connection lost I get "[Failure instance: Traceback (failure with no frames): <class 't wisted.internet.error.ConnectionLost' Connection to the other side was lost in a non-clean fashion.". I am using IP4 TCP sockets. I realize python threads are defective but in this case they are easier to deal with. I will create a chunk of sample code and send it to the list. Thanks, Dan On Sun, Jan 17, 2010 at 6:52 PM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
Here are two tiny apps. They run perfectly on OSX and error like this on windows: Log conenction [Failure instance: Traceback (failure with no frames): <class 't wisted.internet.error.ConnectionLost' Connection to the other side was lost in a non-clean fashion. ] Log conenction [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost' Connection to the other side was lost in a non-clean fashion. .... Start server.py then run tclient.py. Dan On Sun, Jan 17, 2010 at 8:55 PM, Daniel Griffin <dgriff1@gmail.com> wrote:
On Jan 17, 2010, at 10:45 PM, Daniel Griffin wrote:
Ah. This is very simple, and luckily has nothing to do with threads. The simplest explanation is that these errors are showing up because you are printing them out :). If you stop printing them out, they won't show up. More specifically: You generally shouldn't worry about "clean" vs. "non-clean" connection shutdown. There are lots of things that can cause "clean" shutdowns to still lose data, and there are many cases where a "non-clean" shutdown is actually fine. Your application protocol should be framing messages such that you can tell which ones have been responded to. That said, what's going on here is pretty simple. A "clean" shutdown is, broadly speaking, when both ends of the TCP connection agree on where the stream of bytes begins and ends. I send you some data, you read all of it, I shut down the connection, you shut down the connection. If I send you some data, and you *don't* read all of it, then I shut the connection down, the result will be a "non-clean" shutdown, because there is still data outstanding when the connection dies. In this case, you are calling recv(5) in your server, but your client is writing "RESPOND" to its transport. As you can see,
len("RESPOND") 7
"RESPOND" is more than 5 bytes. If you were using Twisted for your server, this wouldn't happen because Twisted always reads all the bytes that are available and delivers them to dataReceived.
Awesome, in my actual code I already implemented something to see if I was "done" with the socket conversation. I was just getting really worried about the "non-clean" messages. Your explanation makes perfect sense. Thanks for the help. Dan On Sun, Jan 17, 2010 at 11:55 PM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
participants (2)
-
Daniel Griffin -
Glyph Lefkowitz