[Python-ideas] Python and Concurrency

[Richard Oudkerk]

On 27/03/07, Josiah Carlson <jcarlson at uci.edu> wrote:
> Those numbers look pretty good.  Would I be correct in assuming that
> there is a speedup sending blocks directly between processes? (though
> perhaps not the 6x that integer sending gains)

Yes, sending blocks directly between processes is over 3 times faster
for 1k blocks, and twice as fast for 4k blocks, but after that it makes
little difference.  (This is using the 'processing.connection'
sub-package which is partly written in C.)

Of course since these blocks are string data you can avoid the pickle
translation which makes things get faster still: the peak bandwidth I
get is 40,000 x 16k blocks / sec = 630 Mb/s.

PS.  It would be nice if the standard library had support for sending
message oriented data over a connection so that you could just do
'recv()' and 'send()' without worrying about whether the whole message
was successfully read/written.  You can use 'socket.makefile()' for
line oriented text messages but not for binary data.