[Python-Dev] Pythonic concurrency

[Christopher Armstrong]

On 10/11/05, Guido van Rossum <guido at python.org> wrote:
> I recall using a non-preemptive system in the past; in Amoeba, to be precise.
>
> Initially it worked great.
>
> But as we added more powerful APIs to the library, we started to run
> into bugs that were just as if you had preemptive scheduling: it
> wouldn't always be predictable whether a call into the library would
> need to do I/O or not (it might use some sort of cache) so it would
> sometimes allow other threads to run and sometimes not. Or a change to
> the library would change this behavior (making a call that didn't use
> to block into sometimes-blocking).

I'm going to be giving a talk at OSDC (in Melbourne) this year about
concurrency systems, and I'm going to talk a lot about the subtleties
between these various non-preemptive (let's call them cooperative :)
systems. I advocate a system that gives you really
straightforward-looking code, but still requires you to annotate the
fact that context switches can occur on every frame where they might
occur (i.e., with a yield). I've given examples before of my new
2.5-yield + twisted Deferred code here, but to recap it just  means
that you have to do:

def foo():
    x = yield getPage()
    return "Yay"

when you want to download a web page, and the caller of 'foo' would
*also* need to do something like "yay = yield foo()". I think this is
a very worthwhile tradeoff for those obsessed with "natural" code.


--
  Twisted   |  Christopher Armstrong: International Man of Twistery
   Radix    |    -- http://radix.twistedmatrix.com
            |  Release Manager, Twisted Project
  \\\V///   |    -- http://twistedmatrix.com
   |o O|    |
w----v----w-+