Sending binary pickled data through TCP

David Hirschfield davidh at ilm.com
Thu Oct 12 21:42:09 EDT 2006


I have a pair of programs which trade python data back and forth by 
pickling up lists of objects on one side (using 
pickle.HIGHEST_PROTOCOL), and sending that data over a TCP socket 
connection to the receiver, who unpickles the data and uses it.

So far this has been working fine, but I now need a way of separating 
multiple chunks of pickled binary data in the stream being sent back and 
forth.

Questions:

Is it safe to do what I'm doing? I didn't think there was anything 
fundamentally wrong with sending binary pickled data, especially in the 
closed, safe environment these programs operate under...but maybe I'm 
making a poor assumption?

I was going to separate the chunks of pickled data with some well-formed 
string, but couldn't that string potentially randomly appear in the 
pickled data? Do I just pick an extremely 
unlikely-to-be-randomly-generated string as the separator? Is there some 
string that will definitely NEVER show up in pickled binary data?

I thought about base64 encoding the data, and then decoding on the 
opposite side (like what xmlrpclib does), but that turns out to be a 
very expensive operation, which I want to avoid, speed is of the essence 
in this situation.

Is there a reliable way to determine the byte count of some pickled 
binary data? Can I rely on len(<pickled data>) == bytes?

Thanks for all responses,
-David

-- 
Presenting:
mediocre nebula.




More information about the Python-list mailing list