[Python-Dev] Cloning threading.py using processes

[Richard Oudkerk]

Version 0.10 of the 'processing' package is available at the cheeseshop:

    http://cheeseshop.python.org/processing

It is intended to make writing programs using processes almost the
same as writing
programs using threads. (By importing from 'processing.dummy' instead
of 'processing'
one can use threads with the same API.)

It has been tested on both Windows and Unix.

Shared objects are created on a 'manager' which runs in its own processes.
Communication with it happens using sockets or (on windows) named pipes.


An example where integers are sent through a shared queue from a child
process to
its parent:

.   from processing import Process, Manager
.
.   def f(q):
.       for i in range(10):
.           q.put(i*i)
.       q.put('STOP')
.
.   if __name__ == '__main__':
.      manager = Manager()
.      queue = manager.Queue(maxsize=10)
.
.      p = Process(target=f, args=[queue])
.      p.start()
.
.      result = None
.      while result != 'STOP':
.           result = queue.get()
.           print result
.
.      p.join()


It has had some changes since the version I posted lasted month:

1) The use of tokens to identify shared objects is now hidden, so now
the API of
'processing' really is very similar to that of 'threading'.

2) It is much faster than before: on both Windows XP and Linux a manager serves
roughly 20,000 requests/second on a 2.5 Ghz Pentium 4.  (Though it is
not a fair comparison that is 50-100 times faster than using
SimpleXMLRPCServer/xmlrpclib.)

3) The manager process just reuses the standard synchronization types from
threading.py, Queue.py and spawns a new thread to serve each process/thread
which owns a proxy.  (The old version was single threaded and had a
select loop.)

4) Registering new shared types is straight forward, for instance

.    from processing.manager import ProcessBaseManager
.
.    class Foo(object):
.        def bar(self):
.            print 'BAR'
.
.    class NewManager(ProcessBaseManager):
.        pass
.
.    NewManager.register('Foo', Foo, exposed=['bar'])
.
.    if __name__ == '__main__':
.        manager = NewManager()
.        foo = manager.Foo()
.        foo.bar()                     # => prints 'BAR'


Cheers

Richard