[Python-ideas] Python and Concurrency

Richard Oudkerk r.m.oudkerk at googlemail.com
Sun Mar 25 23:39:01 CEST 2007


Aahz wrote:
> Maybe that's what it seems to you; to others of us who have been looking
> at this problem for a while, the real question is how to get a better
> multi-process control and IPC library in Python, preferably one that is
> cross-platform.  You can investigate that right now, and you don't even
> need to discuss it with other people.

> (Despite my oft-stated fondness for threading, I do recognize the
> problems with threading, and if there were a way to make processes as
> simple as threads from a programming standpoint, I'd be much more
> willing to push processes.)

<plug>

The processing package at

    http://cheeseshop.python.org/pypi/processing

is multi-platform and mostly follows the API of threading.  It also
allows use of 'shared objects' which live in a manager process.

For example the following code is almost identical to the equivalent
written with threads:

    from processing import Process, Manager

    def f(q):
        for i in range(10):
            q.put(i*i)
        q.put('STOP')

    if __name__ == '__main__':
        manager = Manager()
        queue = manager.Queue(maxsize=3)

        p = Process(target=f, args=[queue])
        p.start()

        result = None
        while result != 'STOP':
            result = queue.get()
            print result

        p.join()

Josiah wrote:
> Without work, #2 isn't "fast" using processes in Python.  It is trivial
> using threads.  But here's the thing: with work, #2 can be made fast.
> Using unix domain sockets (on linux, 3.4 ghz P4 Xeons, DDR2-PC4200
> memory (you can get 50% faster memory nowadays)), I've been able to push
> 400 megs/second between processes.  Maybe anonymous or named pipes, or
> perhaps a shared mmap with some sort of synchronization would allow for
> IPC to be cross platform and just about as fast.

The IPC uses sockets or (on Windows) named pipes.  Linux and Windows
are roughly equal in speed.  On a P4 2.5Ghz laptop one can retreive an
element from a shared dict about 20,000 times/sec.  Not sure if that
qualifies as fast enough.

</plug>

Richard



More information about the Python-ideas mailing list