Sending binary pickled data through TCP
davidh at ilm.com
Fri Oct 13 03:42:09 CEST 2006
I have a pair of programs which trade python data back and forth by
pickling up lists of objects on one side (using
pickle.HIGHEST_PROTOCOL), and sending that data over a TCP socket
connection to the receiver, who unpickles the data and uses it.
So far this has been working fine, but I now need a way of separating
multiple chunks of pickled binary data in the stream being sent back and
Is it safe to do what I'm doing? I didn't think there was anything
fundamentally wrong with sending binary pickled data, especially in the
closed, safe environment these programs operate under...but maybe I'm
making a poor assumption?
I was going to separate the chunks of pickled data with some well-formed
string, but couldn't that string potentially randomly appear in the
pickled data? Do I just pick an extremely
unlikely-to-be-randomly-generated string as the separator? Is there some
string that will definitely NEVER show up in pickled binary data?
I thought about base64 encoding the data, and then decoding on the
opposite side (like what xmlrpclib does), but that turns out to be a
very expensive operation, which I want to avoid, speed is of the essence
in this situation.
Is there a reliable way to determine the byte count of some pickled
binary data? Can I rely on len(<pickled data>) == bytes?
Thanks for all responses,
More information about the Python-list