[Python-Dev] Pythonic concurrency - cooperative MT

[Antoine]

Hi Martin,

[snip]

The "confusion" stems from the fact that two issues are mixed up in this
discussion thread:
- improving concurrency schemes to make it easier to write well-behaving
applications with independent parallel flows
- improving concurrency schemes to improve performance when there are
several hardware threads available

The respective solutions to these problems do not necessarily go hand in
hand.

> To implement that explicitly, you would need an
> asynchronous version of all the functions that may block on
> resources (e.g. file open, socket write, etc.), in order to be
> able to insert a yield statement at that point, after the async
> call, and there should be a way for the scheduler to check if the
> resource is "ready" to be able to put your generator back in the
> runnable queue.

You can also use a helper thread and signal the scheduling loop when some
action in the helper thread has finished. It is an elegant solution
because it helps you keep a small generic scheduling loop instead of
putting select()-like calls in it.
(this is how I've implemented timers in my little cooperative
multi-threading system, for example)

> (A question comes to mind here: Twisted must be doing something
> like this with their "deferred objects", no?  I figure they would
> need to do something like this too.  I will have to check.)

A Deferred object is just the abstraction of a callback - or, rather, two
callbacks: one for success and one for failure. Twisted is architected
around an event loop, which calls your code back when a registered event
happens (for example when an operation is finished, or when some data
arrives on the wire). Compared to generators, it is a different way of
expressing cooperative multi-threading.

Regards

Antoine.