Sending binary pickled data through TCP

MRAB google at mrabarnett.plus.com
Fri Oct 13 14:24:49 CEST 2006


David Hirschfield wrote:
> I have a pair of programs which trade python data back and forth by
> pickling up lists of objects on one side (using
> pickle.HIGHEST_PROTOCOL), and sending that data over a TCP socket
> connection to the receiver, who unpickles the data and uses it.
>
> So far this has been working fine, but I now need a way of separating
> multiple chunks of pickled binary data in the stream being sent back and
> forth.
>
> Questions:
>
> Is it safe to do what I'm doing? I didn't think there was anything
> fundamentally wrong with sending binary pickled data, especially in the
> closed, safe environment these programs operate under...but maybe I'm
> making a poor assumption?
>
> I was going to separate the chunks of pickled data with some well-formed
> string, but couldn't that string potentially randomly appear in the
> pickled data? Do I just pick an extremely
> unlikely-to-be-randomly-generated string as the separator? Is there some
> string that will definitely NEVER show up in pickled binary data?
>
> I thought about base64 encoding the data, and then decoding on the
> opposite side (like what xmlrpclib does), but that turns out to be a
> very expensive operation, which I want to avoid, speed is of the essence
> in this situation.
>
> Is there a reliable way to determine the byte count of some pickled
> binary data? Can I rely on len(<pickled data>) == bytes?
>
Instead of communicating directly with the TCP socket, you could talk
to it via an object which precedes each chunk with a byte count, and if
you're working with multiple streams of picked data, then each chunk
could also have an identifier which specified which stream it belonged
to.




More information about the Python-list mailing list