[Twisted-Python] protocol and transport question
Hi all, I have a problem with a simple client/server communication: I have a server that send the data through "transport" (my class inherit from twisted.internet.protocol.Protocol), every 0.4 seconds. On the client (that now, for test propose are on the same machine) some time I receive two messages concatenated from the server like it send on the same time. Other times, I receive it correctly (in different moments). I tried to debug this situations with print some messages, and I see that twisted sometimes wait "a time" before send data through the channel:
on the server, on twisted.internet.tcp.Connection on writeSomeData I add: print "CCCCCCCCCCCCCCCCC", repr(data), time.time() try: # Limit length of buffer to try to send, because some OSes are too ....
on my code: print "BBBBBBBBBBBBBBBBBBB", repr(msg), time.time() self.transport.write(str(msg))
and I see (then it's wrong): BBBBBBBBBBBBBBBBBBB '230 XXXXXXXXX\n' 1187693294.84 BBBBBBBBBBBBBBBBBBB '235 XXXXXXXXX 144\n' 1187693295.28 CCCCCCCCCCCCCCCCC '230 XXXXXXXXX \n235 XXXXXXXXX 144\n' 1187693295.28
and when it's right: BBBBBBBBBBBBBBBBBBB '230 XXXXXXXXX \n' 1187693282.89 CCCCCCCCCCCCCCCCC '230 XXXXXXXXX \n' 1187693283.06 BBBBBBBBBBBBBBBBBBB '235 XXXXXXXXX 143\n' 1187693283.28 CCCCCCCCCCCCCCCCC '235 XXXXXXXXX 143\n' 1187693283.28
How solve it? Is there a method for say to twisted (transport) to not wait to send the data? (flush the buffer?)
Thanks a lot!
Luca
On Tue, 21 Aug 2007 13:02:34 +0200, Luca Politti luca@unipex.it wrote:
Hi all, I have a problem with a simple client/server communication: I have a server that send the data through "transport" (my class inherit from twisted.internet.protocol.Protocol), every 0.4 seconds. On the client (that now, for test propose are on the same machine) some time I receive two messages concatenated from the server like it send on the same time. Other times, I receive it correctly (in different moments).
For message-oriented protocols (what you seem to be implementing), it is necessary to have some "framing" mechanism - a way to tell where a message ends and the next begins. You can't rely on time to tell messages apart over TCP, since TCP makes few guarantees about when it will do things.
I tried to debug this situations with print some messages, and I see that twisted sometimes wait "a time" before send data through the channel:
Bytes written to a TCP transport will be sent (almost) as soon as they can be. They are not (in the current implementation) sent before the transport.write() call returns, but they will be sent the next time the reactor regains execution control. Unless your application is blocking the reactor from running (ie, performing some long running task), this means the reactor will try to send your data at most a few milliseconds after your write call. It is not necessarily the case that the send will succeed at that time, though. In such a case, the reactor will buffer the data and try to send it again later.
on the server, on twisted.internet.tcp.Connection on writeSomeData I add: print "CCCCCCCCCCCCCCCCC", repr(data), time.time() try: # Limit length of buffer to try to send, because some OSes are too ....
on my code: print "BBBBBBBBBBBBBBBBBBB", repr(msg), time.time() self.transport.write(str(msg))
writeSomeData and transport.write don't necessarily have a one-to-one correspondence to each other, so this debug output might be a bit misleading.
[snip]
How solve it? Is there a method for say to twisted (transport) to not wait to send the data? (flush the buffer?)
Since there's no buffering except when absolutely necessary, there's no way to flush.
So there are three things to watch out for:
* You must use a framing mechanism in order to differentiate your messages. This might be as simple as having them all be the same length, or it might mean including a length prefix (see the NetstringReceiver or Int{8,16,32}StringReceiver protocols in twisted.protocols.base for examples of this), or it might be something more complex.
* Don't block the reactor. If you want to wait a while, use the callLater method of the reactor, not time.sleep. If you have to call a function that will block for a long time before it returns, find an asynchronous version instead, or use the reactor's threadpool.
* Don't call reactor methods from any thread except the one which is running the reactor. This will have unpredictable results and generally be broken.
Jean-Paul
Jean-Paul Calderone ha scritto:
On Tue, 21 Aug 2007 13:02:34 +0200, Luca Politti luca@unipex.it wrote:
Hi all, I have a problem with a simple client/server communication: I have a server that send the data through "transport" (my class inherit from twisted.internet.protocol.Protocol), every 0.4 seconds. On the client (that now, for test propose are on the same machine) some time I receive two messages concatenated from the server like it send on the same time. Other times, I receive it correctly (in different moments).
For message-oriented protocols (what you seem to be implementing), it is necessary to have some "framing" mechanism - a way to tell where a message ends and the next begins. You can't rely on time to tell messages apart over TCP, since TCP makes few guarantees about when it will do things.
I tried to debug this situations with print some messages, and I see that twisted sometimes wait "a time" before send data through the channel:
Bytes written to a TCP transport will be sent (almost) as soon as they can be. They are not (in the current implementation) sent before the transport.write() call returns, but they will be sent the next time the reactor regains execution control. Unless your application is blocking the reactor from running (ie, performing some long running task), this means the reactor will try to send your data at most a few milliseconds after your write call. It is not necessarily the case that the send will succeed at that time, though. In such a case, the reactor will buffer the data and try to send it again later.
on the server, on twisted.internet.tcp.Connection on writeSomeData I add: print "CCCCCCCCCCCCCCCCC", repr(data), time.time() try: # Limit length of buffer to try to send, because some OSes are too ....
on my code: print "BBBBBBBBBBBBBBBBBBB", repr(msg), time.time() self.transport.write(str(msg))
writeSomeData and transport.write don't necessarily have a one-to-one correspondence to each other, so this debug output might be a bit misleading.
[snip]
How solve it? Is there a method for say to twisted (transport) to not wait to send the data? (flush the buffer?)
Since there's no buffering except when absolutely necessary, there's no way to flush.
So there are three things to watch out for:
You must use a framing mechanism in order to differentiate your messages. This might be as simple as having them all be the same length, or it might mean including a length prefix (see the NetstringReceiver or Int{8,16,32}StringReceiver protocols in twisted.protocols.base for examples of this), or it might be something more complex.
Don't block the reactor. If you want to wait a while, use the callLater method of the reactor, not time.sleep. If you have to call a function that will block for a long time before it returns, find an asynchronous version instead, or use the reactor's threadpool.
Don't call reactor methods from any thread except the one which is running the reactor. This will have unpredictable results and generally be broken.
Jean-Paul
Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Ok, Jean-Paul, thanks a lot. We haven't thought to this. We thought that every time we called transport.write, it sent the msg... But we have solved the problem using your first suggest: the framing mechanism :). So now, just before send the msg, I append the length of the msg at the beginning of this. On the other side, when I receive the msg I cut the msg until the length of the msg. The rest is putted in a buffer and processed after. The most important thing is that IT WORKS!!!! :))))
So we thank you very a lot...
Best reguards..
Luca
participants (2)
-
Jean-Paul Calderone -
Luca Politti