[Python-Dev] Simple coroutines?

[Clark C. Evans]

A few definitions:

  iterator: >
    An iterator is an object returned by an __iter__ with
    a next() method which produces values.  An iterator's
    state is kept on the heap (not on the stack), thus
    one could consider it 'resumable'.  An iterator ends when
    it raises an exception, the very special exception,
    'StopIteration' is used to signal that the iterator is
    done producing values, but is not really an error condition.
    
  generator: >
    A syntax for making an iterator, it appears in python
    source code as a function or method with a 'yield'
    statement.  An implementation is free to optimize this
    function, but it acts as if it had created an iterator.
   
  iterlink: >
    An iterlink is an association of one iterator, the consumer,
    to another iterator, the producer.  The iterlink contains
    information about both iterators, and a marker to indicate
    the state of the consumer when it asked for the next value
    of the producer.

  iterstack: > 
    A stack of iterlinks, where the producer of one iterlink is
    the consumer of another.  The top iterator in the iterstack
    is an iterator which is behaving like a producer, but not
    a consumer.  The bottom iterator in the iterstack is one
    which is a producer to a non-iterator.  That is, the caller
    of the bottom iterator is a function or method which is
    not an iterator.

Ok.  With this in mind, I like your idea of a new keyword, 'suspend'.
This keyword could be used by the top iterator in an iterstack.  When
'suspend' happens, the entire iterstack, and its runstate, is setaside.
An a signal is sent to the caller of the bottom iterator; the signal
could be viewed as a 'resumable exception'.   For example, it could
be pictured as a 'SuspendIteration' exception.  More concretely, using a
very simple case.

   class TopIterator:
       """  
            def TopIterator():
                yield "one"
                suspend
                yield "two"
       """
       def _one(self):
           self.next = self._suspend
           return "one"
       def _suspend(self):
           self.next = self._two
           raise SuspendIteration()
       def _two(self):
           self.next = self._stop
           return "two"
       def _stop(self):
           raise StopIteration
       def __iter__():
           self.next = self._one
           return self

    class BottomIterator():
        """
            def BottomIterator():
                producer = iter(TopIterator())
                saved_one = producer.next()
                saved_two = producer.next()
                yield (saved_one,saved_two)
        """
        def _one(self):
            self.saved_one = self.producer.next()
            self.next = self._two
            return self.next()
        def _two(self):
            self.saved_two = self.producer.next()
            self.next = self._stop
            return (self.saved_one, self.saved_two)
       def _stop(self):
            raise StopIteration
        def __iter__(self):
            self.producer = iter(TopIterator())
            self.next = self._one
            return self

    def caller():
        it = iter(BottomIterator())
        while True:
            try:
                 print it.next()
            except StopIteration:
                break
            except SuspendIteration:
                sleep(10)
                continue

Anyway, the behavior probably equivalent to 'resumable exceptions',
however, I don't think it is _different_ item from an iterator, just a
new set of behaviors triggered by the 'suspend' keyword.  As long as the
'iterator construction' is automated, the 'caller()' function can be
delegated to Twisted's reactor, or asynccore or any other cooperative
multitasking base.  The superclasses for SuspendIteration could be
'BlockForSocket', for example.

On Tue, Aug 24, 2004 at 05:57:18PM +1200, Greg Ewing wrote:
| This makes me think that maybe we want another kind of object, similar
| to a generator, but designed to be used for side effects rather than
| to produce values.

  (a) the generation of 'top-level' iterators using 'suspend'
      would be quite easy, see above

  (b) the generation logic for "pass-through" generators (aka
      BottomIterator in the example above) would have to happen
      for the mechanism to be truly useful; SuspendIteration
      would have to be a pass-through exception

  (c) updating itertools to allow SuspendIteration to 'passthrough'

| For want of a better term, let's call it a "cooperator" (so it ends in
| "ator" like "generator" :-). Actually there will be two objects, a
| cooperator-function (analogous to a generator-function) and a
| cooperator-instance (analogous to a generator-iterator).
| 
| Calling a cooperator-function will return a cooperator-instance, which
| will obey the cooperator protocol. This protocol consists of a single
| function run(), which causes the cooperator-instance to perform some
| amount of work and then stop. If there is more work remaining to be
| done, run() returns a true value, otherwise it returns a false value.

Well, run() need not be specified.  It would be implemented by the
lower-level 'reactor' code, aka the "caller" in the example

| There will be a new statement:
| 
|     suspend
| 
| for use inside a cooperator-function. The presence of this statement
| is what distinguishes a cooperator-function from an ordinary function
| (just as the presence of 'yield' distinguishes a generator from an
| ordinary function). Execution of 'suspend' will cause the run() method
| to return with a true value. The next time run() is called, execution
| will resume after the 'suspend'.

Right.  Only that I'd allow both 'yield' and 'suspend' in the same
function, or it really isn't that useful.

| This is really all that's needed, but one further piece of syntax may
| be desirable to make it easier for one cooperator to invoke another:
| 
|     do another_coop()
| 
| would be equivalent to
| 
|     co = another_coop()
|     while co.run():
|         suspend

Exactly; this is where the tough part is, the 'intermediate' cooperators.
Ideally, all generators (since they keep their state on the stack) would
make good 'intermediate' cooperators.

| Something to note is that there's no requirement for a cooperator-
| instance to be implemented by a cooperator-function, just as an
| iterator can be implemented in ways other than a generator.  This
| sidesteps some of the difficulties of mixing Python and C calls, since
| a Python cooperator-function could invoke a cooperator implemented in
| C which in turn invokes another Python cooperator- function, etc. The
| only requirement is that all the objects on the way down obey the
| cooperator protocol.

Yes.  This is the key, you _only_ attempt to suspend iterators, that is
objects that already keep their state in the heap.

| The drawback is that you always have to know when you're calling
| another cooperator and use 'do'. But that's no worse than the current
| situation with generators, where you always have to know you're using
| an iterator and use 'for... yield'.

Er, well, if the interface was an 'SuspendIteration' then it could be
generalized to all iterators.  Afterall, while an exception invalidates
the stack-frame for the current call to next(), it doesn't prevent the
caller from invoking next() once again.

If someone calls next() manually, then SuspendIteration would act as
a regular exception... and be uncaught.

Clark