Sockets: Sending/receiving arbitrary amounts of data

[Neil Schemenauer]

Daniel Klein wrote:
> What if the server doesn't know in advance how much data will
> be sent from the client?

You need to build this into your protocol.  A lot of existing
protocols use terminators to signal the end of data like "\r\n"
or something similar.  I like using Dan Bernstein's netstring
protocol:

    http://cr.yp.to/proto/netstrings.txt

Just an example, if you want to send an arbitrary string the
sender would use:

    def write_string(s, sock):
        sock.send("%lu:" % len(s))
        sock.send(s)
        sock.send(",")
    
    data = "Hello World"
    write_string(data, sock)

reciever:

    def read_string(sock):
        size = ""
        while 1:
            c = sock.recv(1)
            if c == ":":
                break
            elif not c:
                raise IOError, "short netstring read"
            size = size + c
        size = int(size)
        s = sock.recv(size)
        if len(s) != size:
            raise IOError, "short netstring read"
        if sock.recv(1) != ",":
            raise IOError, "missing netstring terminator"
        return s

    data = read_string(sock)
        
Obviously the global "sock" must be an open socket connecting the
two programs.  Some advantages of the netstring protocal is that
it is efficient, especially for large messages.  It makes it much
easier to dynamicly manage memory since the reciever knows in
advance how much data is coming.  It is also 8 bit clean which
means you can send arbirary data and do things like embed
netstrings inside of other netstrings.

  Neil