Pickling over a socket

[Dan Stromberg]

On Tue, Apr 19, 2011 at 11:53 AM, Roger Alexander <rtalexander at mac.com> wrote:
> Hi,
>
> I'm trying to understand how to pickle Python objects over a TCP
> socket.
>
> In the example below (based on code from Foundations of Python Network
> Programming), a client creates a dictionary (lines 34-38) and uses
> pickle.dump at line 42 to write the pickled object using file handle
> make from a socket. The server de-pickles with pickle.load  (line 24),
> again using a file handle made from a socket.
>
> When I run the program, the following output is produced:
>
>    Listening at ('127.0.0.1', 1060)
>    Accepted connection from ('127.0.0.1', 49938)
>    Traceback (most recent call last):
>    File "pickles.py", line 24, in <module>
>        d = pickle.load( s_fh )
>    File "/usr/local/lib/python2.7/pickle.py", line 1378, in load
>        return Unpickler(file).load()
>    File "/usr/local/lib/python2.7/pickle.py", line 857, in load
>        key = read(1)
>    File "/usr/local/lib/python2.7/socket.py", line 380, in read
>        data = self._sock.recv(left)
>    socket.error: [Errno 107] Transport endpoint is not connected
>
> I'm at a loss, can anyone provide any guidance?
>
> Thanks,
>
> Roger Alexander

I played around with it until something worked, and ended up with the
below.  The most significant change was probably using sc.makefile
instead of s.makefile in the server, but I seemed to need some data
framing too despite the pickling.  It's possible you won't need that
if you just flush your file in the client; I don't much pickling
experience - in particular, I don't know if you can concatenate
pickled objects and load them serially from a file-like object without
any (additional) framing.

I like to use bufsock for this sort of stuff, but I'm probably unique
in that.  ^_^   http://stromberg.dnsalias.org/~strombrg/bufsock.html

#!/usr/bin/python

import time
import pickle
import socket, sys
import pprint

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

HOST = sys.argv.pop() if len(sys.argv) == 3 else '127.0.0.1'
PORT = 1060

if sys.argv[1:] == ['server']:

    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    s.bind((HOST, PORT))
    s.listen(1)

    print 'Listening at', s.getsockname()

    sc, sockname = s.accept()

    print 'Accepted connection from', sockname

    sf = sc.makefile( "rb" )

    length_list = []
    while True:
        char = sf.read(1)
        if char == '\n':
            break
        else:
            length_list.append(int(char))
    length = 0
    for digit in length_list:
        length = length * 10 + digit
    data = sf.read(length)

    d = pickle.loads(data)

    pprint.pprint(d)

    sc.shutdown(socket.SHUT_RDWR)
    sc.close()
    s.close()

elif sys.argv[1:] == ['client']:

    s.connect((HOST, PORT))
    # s.shutdown(socket.SHUT_RD)

    d = dict()

    d[ 'Name' ]     = 'Jake Thompson.'
    d[ 'Age' ]      = 25
    d[ 'Location' ] = 'Washington, D.C.'

    sf = s.makefile( "wb" )

    string = pickle.dumps( d, pickle.HIGHEST_PROTOCOL )
    sf.write('%d\n' % len(string))
    sf.write(string)
    sf.flush()

    #time.sleep(10)
    sf.close()
    s.shutdown(socket.SHUT_RDWR)
    # s.close()

else:
    print >>sys.stderr, 'usage: streamer.py server|client [host]'