Problem with multiprocessing

[Dan Stromberg]

Share as little as possible between your various processes - shared, mutable
state is a parallelism tragedy.

If you can avoid sharing an entire dictionary, do so.  It'd probably be
better to dedicate one process to updating your dictionary, and then using a
multiprocessing.Queue to pass delta records from your workers to your
dictionary management process.

Also, I'm inclined to doubt it's going to work well to have multiple
processes doing I/O on the same socket - you'd probably best have a process
that does all the I/O on the socket, and then, again, have one or more
multprocessing.Queue's that pass I/O results/requests around.

On Thu, May 19, 2011 at 6:10 AM, Pietro Abate
<Pietro.Abate at pps.jussieu.fr>wrote:

> Hi all,
>
> I'm a bit struggling to understand a KeyError raised by the multiprocessing
> library.
>
> My idea is pretty simple. I want to create a server that will spawn a
> number of
> workers that will share the same socket and handle requests independently.
>  The
> goal is to build a 3-tier structure where all requests are handled via an
> http
> server and then dispatched to nodes sitting in a cluster and from nodes to
> workers via the multiprocessing managers...
>
> There is one public server, one node per machine and x number of workers on
> each machine depending on the number of cores... I know I can use a more
> sophisticated library, but for such a simple task (I'm just prototyping
> here) I
> would just use the multiprocessing library... Is this possible or I should
> explore directly other solutions ? I feel I'm very close to have something
> working here ...
>
> The problem with the code below is that if I run the server as
> `python server.py 1` , that is, using only one process, it works as
> expected.
>
> However if I spawn two processes (`python server.py 2`) listening for
> connections, I get a nasty error :
>
>    $python client.py ping
>    Traceback (most recent call last):
>      File "client.py", line 24, in <module>
>    sys.exit(main(sys.argv))
>      File "client.py", line 21, in main
>        print m.solver(args[1])._getvalue()
>      File "/usr/lib/python2.6/multiprocessing/managers.py", line 637, in
> temp
>        authkey=self._authkey, exposed=exp
>      File "/usr/lib/python2.6/multiprocessing/managers.py", line 894, in
> AutoProxy
>        incref=incref)
>      File "/usr/lib/python2.6/multiprocessing/managers.py", line 700, in
> __init__
>        self._incref()
>      File "/usr/lib/python2.6/multiprocessing/managers.py", line 750, in
> _incref
>        dispatch(conn, None, 'incref', (self._id,))
>      File "/usr/lib/python2.6/multiprocessing/managers.py", line 79, in
> dispatch
>        raise convert_to_error(kind, result)
>    multiprocessing.managers.RemoteError:
>
>  ---------------------------------------------------------------------------
>    Traceback (most recent call last):
>      File "/usr/lib/python2.6/multiprocessing/managers.py", line 181, in
> handle_request
>        result = func(c, *args, **kwds)
>      File "/usr/lib/python2.6/multiprocessing/managers.py", line 402, in
> incref
>        self.id_to_refcount[ident] += 1
>    KeyError: '7fb51084c518'
>
>  ---------------------------------------------------------------------------
>
> My understanding is that all processes share the same socket (from the
> Manager). When a client wants to connect, a new connection is created and
> server independently by that process. If you look at the server trace
> (using
> logging), it actually receives the connection, handles it, but fails to
> communicate back to the client.
>
> Can anybody shed some light for me and maybe propose a solution ?
>
> thanks
> pietro
>
> ----------------------------------------
>
> Server :
>
>    import sys
>    from multiprocessing.managers import BaseManager, BaseProxy, Process
>
>    def baz(aa) :
>        l = []
>        for i in range(3) :
>          l.append(aa)
>        return l
>
>    class SolverManager(BaseManager): pass
>
>    class MyProxy(BaseProxy): pass
>
>    manager = SolverManager(address=('127.0.0.1', 50000), authkey='mpm')
>    manager.register('solver', callable=baz, proxytype=MyProxy)
>
>    def serve_forever(server):
>        try :
>            server.serve_forever()
>        except KeyboardInterrupt:
>            pass
>
>    def runpool(n):
>        server = manager.get_server()
>        workers = []
>
>        for i in range(int(n)):
>            Process(target=serve_forever, args=(server,)).start()
>
>    if __name__ == '__main__':
>        runpool(sys.argv[1])
>
>
> Client :
>
>    import sys
>    from multiprocessing.managers import BaseManager, BaseProxy
>
>    import multiprocessing, logging
>
>    class SolverManager(BaseManager): pass
>
>    class MyProxy(BaseProxy): pass
>
>    def main(args) :
>        SolverManager.register('solver')
>        m = SolverManager(address=('127.0.0.1', 50000), authkey='mpm')
>        m.connect()
>
>        print m.solver(args[1])._getvalue()
>
>    if __name__ == '__main__':
>        sys.exit(main(sys.argv))
>
>
> also tried on stack overflow and the python list, but I didn't manage to
> come up
> with a working solution yet...
>
> --
> ----
> http://en.wikipedia.org/wiki/Posting_style
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20110519/36a8f520/attachment.html>