[New-bugs-announce] [issue11119] Passing a socket to a process (multiprocessing module)

[pen hill]

New submission from pen hill <imbenhill at gmail.com>:

When I run the following listing (server_multi.py) by using multiprocessing module of python, it runs successfully on a Linux machine. However, on a 64-bit windows machine I get the following error:

pickle.PicklingError: Can't pickle <built-in method recvfrom_into of _socket.socket object at 0x0000000002D2CF10>: it's not found as __main__.recvfrom_into

This module is from the book Foundations of Python Network Programming, 2nd Edition. I am pasting the code with the permission of the author. It seems that socket "listen_sock" cannot be pickled under 64-bit
Windows, however it can be pickled under Linux. How can we remove this issue on Windows? It is interesting that if I use the threading module, it works for both Windows and Linux. Thanks. 

server_multi.py :
#usage: $ python server_multi.py localhost process
#!/usr/bin/env python
# Foundations of Python Network Programming - Chapter 7 - server_multi.py
# Using multiple threads or processes to serve several clients in parallel.

import sys, time, lancelot
from multiprocessing import Process
from server_simple import server_loop
from threading import Thread

WORKER_CLASSES = {'thread': Thread, 'process': Process}
WORKER_MAX = 10

def start_worker(Worker, listen_sock):
    worker = Worker(target=server_loop, args=(listen_sock,))
    worker.daemon = True  # exit when the main process does
    worker.start()
    return worker

if __name__ == '__main__':
    if len(sys.argv) != 3 or sys.argv[2] not in WORKER_CLASSES:
        print >>sys.stderr, 'usage: server_multi.py interface thread|process'
        sys.exit(2)
    Worker = WORKER_CLASSES[sys.argv.pop()]  # setup() wants len(argv)==2

    # Every worker will accept() forever on the same listening socket.

    listen_sock = lancelot.setup()
    workers = []
    for i in range(WORKER_MAX):
        workers.append(start_worker(Worker, listen_sock))

    # Check every two seconds for dead workers, and replace them.

    while True:
        time.sleep(2)
        for worker in workers:
            if not worker.is_alive():
                print worker.name, "died; starting replacement worker"
                workers.remove(worker)
                workers.append(start_worker(Worker, listen_sock))

lancelot.py:

#!/usr/bin/env python
# Foundations of Python Network Programming - Chapter 7 - lancelot.py
# Constants and routines for supporting a certain network conversation.

import socket, sys

PORT = 1060
qa = (('What is your name?', 'My name is Sir Lancelot of Camelot.'),
      ('What is your quest?', 'To seek the Holy Grail.'),
      ('What is your favorite color?', 'Blue.'))
qadict = dict(qa)

def recv_until(sock, suffix):
    message = ''
    while not message.endswith(suffix):
        data = sock.recv(4096)
        if not data:
            raise EOFError('socket closed before we saw %r' % suffix)
        message += data
    return message

def setup():
    if len(sys.argv) != 2:
        print >>sys.stderr, 'usage: %s interface' % sys.argv[0]
        exit(2)
    interface = sys.argv[1]
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    sock.bind((interface, PORT))
    sock.listen(128)
    print 'Ready and listening at %r port %d' % (interface, PORT)
    return sock

server_simple.py:

#!/usr/bin/env python
# Foundations of Python Network Programming - Chapter 7 - server_simple.py
# Simple server that only serves one client at a time; others have to wait.

import lancelot

def handle_client(client_sock):
    try:
        while True:
            question = lancelot.recv_until(client_sock, '?')
            answer = lancelot.qadict[question]
            client_sock.sendall(answer)
    except EOFError:
        client_sock.close()

def server_loop(listen_sock):
    while True:
        client_sock, sockname = listen_sock.accept()
        handle_client(client_sock)

if __name__ == '__main__':
    listen_sock = lancelot.setup()
    server_loop(listen_sock)

----------
components: None
messages: 127922
nosy: pen hill
priority: normal
severity: normal
status: open
title: Passing a socket to a process (multiprocessing module)
type: crash
versions: Python 2.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue11119>
_______________________________________