Proposal for adding Shallow Threads and a Main Loop to Python

Rhamphoryncus rhamph at
Fri Mar 18 16:45:19 CET 2005

Diez B. Roggisch wrote:
> Hi,
> a few questions:
> > A shallow thread is just a generator modified in the most obvious
> > possible.  The yield statement is replaced with a waitfor
> > You give it the object you wish to "wait for".  Then when it's
> > you get back a return value or an exception.  These waitfor
> > are the only points where your shallow thread may get suspended, so
> > it's always explicit.  If you call another function it will be
> > exactly as a normal function call from a generator (this is why
> > 'shallow'), so there's no issues with calling C functions.
> >
> > On the other end of things, your shallow thread object would have
> > __resume__ and __resumeexc__ methods (or perhaps a single
> > method with a default raise_=False argument).  They return a tuple
> > (mode,value) where mode is a string of 'waitfor', 'exception', or
> > 'return' and value corresponds to that.  These methods will be used
> > a main loop to resume the shallow thread, like next() is used with
> > generator.
> So basically a shallow thread could be written in today's python like
> class ShallowThread(object):
>     def __init__(self, data=None):
> = data
>     def run(self, initial_data=None):
>         while True:
>             ....
>             yield <some_state>
>             new_data =
>             ....
> st = ShallowThread().run(<some_initial_data>)
> while True:
>     result =
> = <some_subsequent_data>
> Basically <some_state> is your above mentioned tuple, and waitfor is
> yield with a return value that I modeled with explicit state called
> Is that correct so far?

Yes, that's the general idea.  I would give the following example
however, using Twisted 2.0's deferredGenerator facility (and assuming
all the functions I depend on are still modified the same way, just for
a fair comparison).

from twisted.internet.defer import *
from twisted.internet import reactor

def get_and_save(path):
    thingy = waitForDeferred(urllib.urlopen(path, async=True))
    yield thingy
    infile = thingy.getResult()
    thingy = waitForDeferred(open(split('/')[-1], async=True)
    yield thingy
    outfile = thingy.getResult()
    thingy = waitForDeferred(
    yield thingy
    data = thingy.getResult()
    thingy = waitForDeferred(outfile.write(data, async=True))
    yield thingy
    thingy.getResult()  # Still needed to raise exceptions
get_and_save = deferredGenerator(get_and_save)

def main():
    a = get_and_save("")
    b = get_and_save("")
    c = get_and_save("")

    thingy = waitForDeferred(allDone(a, b, c))
    yield thingy
main = deferredGenerator(main)

if __name__ == "__main__":
    d = main()
    d.addBoth(lambda _: reactor.stop())

> > import mainloop, urllib
> >
> > def get_and_save(path):
> >     infile = waitfor urllib.urlopen(path, async=True)
> >     outfile = waitfor open(path.split('/')[-1], async=True)
> >     waitfor outfile.write(waitfor,
> >     infile.close()
> >     outfile.close()
> >
> > def main():
> >     a = get_and_save("")
> >     b = get_and_save("")
> >     c =
> >
> >     waitfor allDone(a, b, c)
> >
> > if __name__ == "__main__":
> >     mainloop.runUntil(main())
> >
> > Well there you have it.  I've glossed over many details but they
can be
> > cleared up later.  What I need to know now is how everybody else
> > about it.  Is this something you would use?  Does it seem like the
> > right way to do it?  And of course the all important one, can I get
> > in to python core?  <0.5 wink>
> I've difficulties grasping where the actual work is done - the event
> notifier thingies are sort of generators themselves, and the mainloop
> them and calls some execute method on them?

It depends.  For files, the mainloop would have something like a
select-based polling loop.  Once something is ready, it would set the
appropriate event notifier to the "done" state and notify everything
waiting on it.

> And now the final $1000,0000.00 question - why all this? No offense
> - it's a little bit more comfortable than the generators approach
> by others (e.g. David Mertz if I recall corretly) - but to my view,
> _could_ be done in today python because we have generators. Or not?

Go and compare the two get_and_save functions.  Would you call that a
"little bit more comfortable"?  Now imagine the function was 20 lines
to begin with, involving a couple loops and some list comprehensions.
Then just for kicks, imagine it without generators, instead turning it
into a state machine.

Yes, you can *technically* do it using generators (and they're what
make the implementation easy).  But you could also do it using a state
machine.  Doesn't mean it's practical though.

Adam Olsen, aka Rhamphoryncus

More information about the Python-list mailing list