[Python-Dev] Pythonic concurrency

[Jim Jewett]

Bruce Eckel wrote:

> 3) Tasks are cheap enough that I can make
> thousands of them, ...

> 4) Tasks are "self-guarding," so they prevent
> other tasks from interfering with them. The
> only way tasks can communicate with each
> other is through some kind of formal
> mechanism (something queue-ish,
> I'd imagine).

I think these two are the hardest to reconcile.

Shane Hathaway's suggestion starts from the
process end.

A new process isn't cheap.  Keeping the other
process' interpreter alive and feeding it more
requests through a queue just hides the
problem; you can't have more non-sequential
tasks than processors without restarting the
whole contention issue.  Even using sequential
tasks (similar to "import dummy_thread") lets
task1 mess up the builtins (or other library
modules) for future task2.  The more guards
you add, the heavier each task gets.

At the other end are generators; I think what
generators are missing is

(A)  You can't easily send them a message.

This can be solved by wrapping them in an
object, or (probably) by waiting until 2.5.

(B)  The programmer has to supply a scheduler.

This could be solved by a standard library module.

(C)  That scheduler is non-preemptive.  A single
greedy generator can starve all the others.

You can reduce the problem by scheduling
generators on more than one thread.

To really solve it would require language support.
That *might* be almost as simple as a new object
type that automatically yielded control every so often
(like threads).

(D)  Generators can interfere with each other unless
the programmer is careful to treat all externally visible
objects as immutable.

Again, it would require language support.  I also have
a vague feeling that fixing (C) or (D) might make the
other worse.

(E)  Generators are not reentrant.

I know you said that some restrictions are reasonable,
and this might fall into that category ... but I think this
becomes a problem as soon as Tasks can handle
more than one type of message, or can send delayed
replies to specific correspondents.  (To finish your
request, I need some more information...)

-jJ