[Python-Dev] Pythonic concurrency

[Josiah Carlson]

Michael Sparks <ms at cerenity.org> wrote:
> 
> On Thursday 06 October 2005 23:15, Josiah Carlson wrote:
> [... 6 specific use cases ...]
> > If Kamaelia is able to handle all of the above mechanisms in both a
> > blocking and non-blocking fashion, then I would guess it has the basic
> > requirements for most concurrent applications.
> 
> It can. I can easily knock up examples for each if required :-)

That's cool, I trust you.  One thing I notice is absent from the
Kamaelia page is benchmarks.

On the one hand, benchmarks are technically useless, as one can tend to
benchmark those things that a system does well, and ignore those things
that it does poorly (take, for example how PyLinda's speed test only
ever inserts and removes one tuple at a time...try inserting 100k  and
use wildcards to extract those 100k, and you'll note how poor it
performs, or database benchmarks, etc.).  However, if one's benchmarks
provide examples from real use, then it shows that at least someone has
gotten some X performance from the system.

I'm personally interested in latency and throughput for varying sizes of
data being passed through the system.


> That said, a more interesting example implemented this week (as part of
> a rapid prototyping project to look at collaborative community radio)
> implements an networked audio mixer matrix. That allows mutiple sources of
> audio to be mixed, sent on to multiple destinations, may be duplicate mixes
> of each other, but also may select different mixes. The same system also
> includes point to point communications for network control of the mix.

Very neat.  How much data?  What kind of throughput?  What kinds of
latencies?

> That application covers ( I /think/ ) 1, 2, 3, 4,  and 6 on your list of
> things as I understand what you mean. 5 is fairly trivial though.

Cool.

> Regarding blocking & non-blocking, links can be marked to synchronous, which
> forces blocking style behaviour. Since generally we're using generators, we
> can't block for real which is why we throw an exception there. However,
> threaded components can & do block. The reason for this was due to the
> architecture being inspired by noting the similarities between asynchronous
> hardware systems/langages and network systems.

On the client side, I was lazy and used synchronous/blocking sockets to
block on read/write (every client thread gets its own connection,
meaning that tuple puts are never sitting in a queue).  I've also got
server-side timeouts for when you don't want to wait too long for data.
    rslt = tplspace.get(PATTERN, timeout=None)


> > into my tuple space implementation before it is released. 
> 
> I'd be interested in hearing more about that BTW. One thing we've found is
> that much organic systems have a neural system for communications between
> things, (hence Axon :), that you also need to equivalent of a hormonal system.
> In the unix shell world, IMO the environment acts as that for pipelines, and
> similarly that's why we have an assistant system. (Which has key/value lookup
> facilities)

I have two recent posts about the performance and features of a (hacked
together) tuple space system I worked on (for two afternoons) in my blog.
"Feel Lucky" for "Josiah Carlson" in google and you will find it.


> It's a less obvious requirement, but is a useful one nonetheless, so I don't
> really see a message passing style as excluding a linda approach - since
> they're orthoganal approaches.

Indeed.  For me, the idea of being able to toss a tuple into memory
somewhere and being able to find it later maps into my mind as:
('name', arg1, ...) -> name(arg1, ...), which is, quite literally, an
RPC semantic (which seems a bit more natural to me than subscribing to
the 'name' queue).  With the ability to send to either single or
multiple listeners, you get message passing, broadcast messages, and a
standard job/result queueing semantic. The only thing that it is missing
is a prioritization mechanism (fifo, numeric priority, etc.), which
would get us a job scheduling kernel. Not bad for a "message
passing"/"tuple space"/"IPC" library.  (all of the above described have
direct algorithms for implementation).


 - Josiah