Pickled objects over the network

[Jean-Paul Calderone]

On Tue, 17 Jul 2007 14:57:16 -0700, Walker Lindley <brotherjenos at gmail.com> wrote:
>I'm working on a distributed computing program and need to send Python
>objects over a TCP socket. Specifically, the objects that I'm working with
>subclass the builtin list type (I don't know whether or not that matters),
>but also include other data fields. These objects are put into dictionaries
>along with some other control information and the dictionaries are pickled
>and sent across the network. I'm using the makefile() function in socket to
>get a file descriptor for the socket and then calling pickle.dump with the
>dictionary object and the socket file descriptor. I was originally using
>pickle protocol 0, but then read about the possibility of buffering problems
>and realized that protocol 0 was giving me strings about 1500 characters
>long. After some testing, I settled on protocol 2 which manages to package
>the data into about 400 characters.
>
>In any case, the client side of the program appears to send the data just
>fine, at least as far as I can tell. On the server side, however, the data
>takes a very long time to get there and then I get an error in the pickle
>module:
>Traceback (most recent call last):
>  File "server.py", line 176, in <module>
>    serv.runServer()
>  File "server.py", line 99, in runServer
>    recvPacket = pickle.load(clientFile)
>  File "/usr/lib/python2.5/pickle.py", line 1370, in load
>    return Unpickler(file).load()
>  File "/usr/lib/python2.5/pickle.py", line 858, in load
>    dispatch[key](self)
>  File "/usr/lib/python2.5/pickle.py", line 1187, in load_appends
>    mark = self.marker()
>  File "/usr/lib/python2.5/pickle.py", line 874, in marker
>    while stack[k] is not mark: k = k-1
>IndexError: list index out of range
>
>Hopefully I'm doing something obviously wrong, but if anyone can help based
>on that description or if you need to see the source, please let me know
>(it's GPL'd). Thank you so much for any help.
>

The obvious thing you're doing wrong is using pickle over a network. ;)

  http://jcalderone.livejournal.com/15864.html

Jean-Paul