[Python-ideas] Python and Concurrency
jcarlson at uci.edu
Mon Mar 26 01:15:51 CEST 2007
"Richard Oudkerk" <r.m.oudkerk at googlemail.com> wrote:
> Josiah wrote:
> > Without work, #2 isn't "fast" using processes in Python. It is trivial
> > using threads. But here's the thing: with work, #2 can be made fast.
> > Using unix domain sockets (on linux, 3.4 ghz P4 Xeons, DDR2-PC4200
> > memory (you can get 50% faster memory nowadays)), I've been able to push
> > 400 megs/second between processes. Maybe anonymous or named pipes, or
> > perhaps a shared mmap with some sort of synchronization would allow for
> > IPC to be cross platform and just about as fast.
> The IPC uses sockets or (on Windows) named pipes. Linux and Windows
> are roughly equal in speed. On a P4 2.5Ghz laptop one can retreive an
> element from a shared dict about 20,000 times/sec. Not sure if that
> qualifies as fast enough.
Depends on what the element is, but I suspect it isn't fast enough.
Fairly large native dictionaries seem to run on the order of 1.3 million
fetches/second on my 2.8 ghz machine.
>>> import time
>>> d = dict.fromkeys(xrange(65536))
>>> if 1:
... t = time.time()
... for j in xrange(1000000):
... _ = d[j&65535]
... print 1000000/(time.time()-t)
But really, transferring little bits of data back and forth isn't what
is of my concern in terms of speed. My real concern is transferring
nontrivial blocks of data; I usually benchmark blocks of sizes: 1k, 4k,
16k, 64k, 256k, 1M, 4M, 16M, and 64M. Those are usually pretty good to
discover the "sweet spot" for a particular implementation, and also
allow a person to discover whether or not their system can be used for
nontrivial processor loads.
More information about the Python-ideas