"Richard Oudkerk" <r.m.oudkerk@googlemail.com> wrote:
Josiah wrote:
Without work, #2 isn't "fast" using processes in Python. It is trivial using threads. But here's the thing: with work, #2 can be made fast. Using unix domain sockets (on linux, 3.4 ghz P4 Xeons, DDR2-PC4200 memory (you can get 50% faster memory nowadays)), I've been able to push 400 megs/second between processes. Maybe anonymous or named pipes, or perhaps a shared mmap with some sort of synchronization would allow for IPC to be cross platform and just about as fast.
The IPC uses sockets or (on Windows) named pipes. Linux and Windows are roughly equal in speed. On a P4 2.5Ghz laptop one can retreive an element from a shared dict about 20,000 times/sec. Not sure if that qualifies as fast enough.
Depends on what the element is, but I suspect it isn't fast enough. Fairly large native dictionaries seem to run on the order of 1.3 million fetches/second on my 2.8 ghz machine. >>> import time >>> d = dict.fromkeys(xrange(65536)) >>> if 1: ... t = time.time() ... for j in xrange(1000000): ... _ = d[j&65535] ... print 1000000/(time.time()-t) ... 1305482.97346 >>> But really, transferring little bits of data back and forth isn't what is of my concern in terms of speed. My real concern is transferring nontrivial blocks of data; I usually benchmark blocks of sizes: 1k, 4k, 16k, 64k, 256k, 1M, 4M, 16M, and 64M. Those are usually pretty good to discover the "sweet spot" for a particular implementation, and also allow a person to discover whether or not their system can be used for nontrivial processor loads. - Josiah