[Python-Dev] Pythonic concurrency - cooperative MT

[Christopher Armstrong]

On 10/2/05, Martin Blais <blais at furius.ca> wrote:
> One of the problems that you have with using generators like
> this, is that automatic "yield" on resource access does not occur
> automatically, like it does in threading.  With threads, the
> kernel is invoked when access to a low-level resource is
> requested, and may decide to put your process in the wait queue
> when it judges necessary.  I don't know how you would do that
> with generators.  To implement that explicitly, you would need an
> asynchronous version of all the functions that may block on
> resources (e.g. file open, socket write, etc.), in order to be
> able to insert a yield statement at that point, after the async
> call, and there should be a way for the scheduler to check if the
> resource is "ready" to be able to put your generator back in the
> runnable queue.
>
> (A question comes to mind here: Twisted must be doing something
> like this with their "deferred objects", no?  I figure they would
> need to do something like this too.  I will have to check.)

As I mentioned in the predecessor of this thread (I think), I've
written a thing called "Defgen" or "Deferred Generators" which allows
you to write a generator to yield control when waiting for a Deferred
to fire. So this is basically "yield or resource access". In the
Twisted universe, every asynchronous resource-retrieval is done by
returning a Deferred and later firing that Deferred. Generally, you
add callbacks to get the value, but if you use defgen you can say
stuff like (in Python 2.5 syntax)
try:
    x = yield getPage('http://python.org/')
except PageNotFound:
    print "Where did Python go!"
else:
    assert "object-oriented" in x

Many in the Twisted community get itchy about over-use of defgen,
since it makes it easier to assume too much consistency in state, but
it's still light-years beyond pre-emptive shared-memory threading when
it comes to that.

--
  Twisted   |  Christopher Armstrong: International Man of Twistery
   Radix    |    -- http://radix.twistedmatrix.com
            |  Release Manager, Twisted Project
  \\\V///   |    -- http://twistedmatrix.com
   |o O|    |
w----v----w-+