Designing socket messaging format

[Chris Tavares]

"Stephen" <shriek at gmx.co.uk> wrote in message
news:97ae44ee.0111111912.14006c7f at posting.google.com...
> I'm developing a program which sends messages back and forth
> between two servers using sockets.  The message format I've
> been given is "YYYYTTTTTTTTTTTTTTTTTTTTTTT"
>
> Where ~
> Y is the message type, which in turn affects the message length.
> T represents a text message. (eg. "Hello World"). Note that
>   it could also contain newlines (eg. "Hello World\nGood bye")
>
>
>
> This was easy to build ontop of SocketServer ~
>
> import SocketServer
>
> class RequestServer(SocketServer.ThreadingTCPServer):
> allow_reuse_address = 1
>
> class RequestHandler(SocketServer.StreamRequestHandler):
> def handle(self):
> request = self.rfile.readline()
> print "Received message, %s" % request
> self.wfile.write("Received OK")
> print "Ending request"
>
> host = "127.0.0.1"
> port = 1111
> server = RequestServer((host, port), RequestHandler)
> print "SERVER : listening to socket [%s:%s]" % (host, port)
> server.serve_forever()
>
> However, the readline() makes it necessary to send a "\n" at
> the end of the message. So, the message format becomes
> "YYYYTTTTTTTTTTTTTTTTTTTTTTT\n"
>
> This introduces the problem that if any of the text
> characters (T) are a "\n", the message will be terminated
> too soon.
>
>
> A previous post to CLP provided the following solution
> which looks out for a "\r\n" message termination string ~
>
> import SocketServer
>
> class RequestServer(SocketServer.ThreadingTCPServer):
> allow_reuse_address = 1
>
> class RequestHandler(SocketServer.StreamRequestHandler):
> def handle(self):
> request = ''
> while 1:
> line = self.rfile.readline()
> if line in (None, "\n", "\r\n", ""):
> break
> request = request + line
> self.server.request = request
> request = string.rstrip(request)
> print "Received message, %s" % request
> self.wfile.write("Received OK")
>
> host = "127.0.0.1"
> port = 1111
> server = RequestServer((host, port), RequestHandler)
> print "SERVER : listening to socket [%s:%s]" % (host, port)
> server.serve_forever()
>
>
> So, should the message format change to ~
>
> YYYY\n
> TTTTTTTTTTTTTTTTTTTTTTT\n
> \r\n
>
> Other than facilitating use of SocketServer.ThreadingTCPServer,
> it seems that asyncore also looks out for termination strings,
> and the "Python Sockets HowTo" infers that using fixed-length
> for message termination is a bad idea.  So everything seems to
> be pointing this way.
>
> Am I correct in my understanding ?
>
> Stephen
>

Welcome to the wonderful world of ambiguous message formats!

The typical solution is to prefix your message with a length packet, and
forget about terminating characters. So, your packet would look something
like this:

LLLLYYYYTTTTTTTTTTTTTTTT

Where: LLLL = total length of this message in bytes

You would read in your message length, then read that many more bytes off
the socket.

-Chris