[Python-ideas] x=(yield from) confusion [was:Yet another alternative name for yield-from]

Jacob Holm jh at improva.dk
Tue Apr 7 17:00:27 CEST 2009


Greg Ewing wrote:
> Jacob Holm wrote:
>
>> One major reason for factoring something out of a coroutine would be 
>> if the factored-out code was independently useful as a coroutine.
>
> So provide another entry point for using that part
> as a top level, or manually apply the wrapper when
> it's appropriate.

It is inconvenient to have to keep separate versions around, and it 
doesn't solve the problem.  See example below.

>
> I find it extroardinary that people seem to have
> latched onto David Beazley's idiosyncratic definition
> of a "coroutine" and decided that it's written on
> a stone tablet from Mt. Sinai that we must always
> wrap them in his decorator.

I find it extraordinary that my critical perspective on this issue makes 
you think I am taking his tutorial as dogma.  There is a real issue here 
IMNSHO, and the @coroutine examples just happens to be the easiest way I 
can see of explaining it.

>
>> @coroutine
>> def avg_diff():
>>    a = yield from avg2() start None  # "start EXPR" means use EXPR 
>> for first value to yield instead of next()
>>    b = yield from avg2() start a     # "start EXPR" means use EXPR 
>> for first value to yield instead of next()
>
> I'm going to need a less abstract example to see
> why you might want to do something like that.
>

Ok,  below you will find a modified version of your parser example taken 
from 
http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/parser.txt

The modification consists of applying the @coroutine decorator to 
parse_items and parse_elem and changing them to yield a stream of 
2-tuples describing how each token sent to the coroutine was interpreted.

The expected output is:

Feeding: '<foo>'
Yielding: ('Open', 'foo')
Feeding: 'This'
Yielding: ('Data', 'This')
Feeding: 'is'
Yielding: ('Data', 'is')
Feeding: 'a'
Yielding: ('Data', 'a')
Feeding: '<b>'
Yielding: ('Open', 'b')
Feeding: 'foo'
Yielding: ('Data', 'foo')
Feeding: 'file'
Yielding: ('Data', 'file')
Feeding: '</b>'
Yielding: ('Close', 'b')
Feeding: 'you'
Yielding: ('Data', 'you')
Feeding: 'know.'
Yielding: ('Data', 'know.')
Feeding: '</foo>'
Yielding: ('Close', 'foo')
[('foo', ['This', 'is', 'a', ('b', ['foo', 'file']), 'you', 'know.'])]


I can't see a nice way to get the same sequence of send() calls to yield 
the same values without the ability to override the way the value for 
the initial yield in yield-from is computed.  Even avoiding the use of 
@coroutine, I will still need to pass extra arguments to the generator 
functions to control the initial yield.

- Jacob
------------------------------------------------------------------------

# Support code from example at http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/parser.txt

import re
pat = re.compile(r"(\S+)|(<[^>]*>)")
  
def scanner(text):
    for m in pat.finditer(text):
        token = m.group(0)
        print "Feeding:", repr(token)
        yield token
    yield None # to signal EOF
  
text = "<foo> This is a <b> foo file </b> you know. </foo>"
token_stream = scanner(text)

def is_opening_tag(token):
    return token.startswith("<") and not token.startswith("</")


# Coroutine decorator copied from earlier mail, based on the one in http://dabeaz.com/coroutines/

def coroutine(func):
    def start(*args, **kwargs):
        cr = func(*args, **kwargs)
        v = cr.next()
        if v is not None:
            raise RuntimeError('first yield from coroutine was not None')
        return cr
    return start


# Runner modified from example at http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/parser.txt
# to also print the yielded values.

def run():
    parser = parse_items()
    # The original forgot to call next() here.  That is not necessary in this version since parse_items uses
    # the @coroutine decorator.
    try:
        for m in pat.finditer(text):
            token = m.group(0)
            print "Feeding:", repr(token)
            v = parser.send(token)
            print "Yielded:", v
        parser.send(None) # to signal EOF
    except StopIteration, e:
        tree = e.args[0]
        print tree


# parse_elem modified from example at http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/parser.txt
# to make it a coroutine and to yield a sequence of 2-tuples describing how the recieved data was interpreted.
# (Does not yield the final ('Close', <tagname>) because it returns the tree instead). 

@coroutine
def parse_elem():
    opening_tag = yield
    name = opening_tag[1:-1]
    closing_tag = "</%s>" % name
    items = yield from parse_items(closing_tag) start ('Open', name)
    return (name, items)


# parse_items modified from example at http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/parser.txt
# to make it a coroutine and to yield a sequence of 2-tuples describing how the recieved data was interpreted.

@coroutine
def parse_items(closing_tag = None):
    elems = []
    token = yield
    while token != closing_tag:
        if is_opening_tag(token):
            subtree = yield from parse_elem() as p start p.send(token)
            elems.append(subtree)
            out = ('Close', subtree[0])
        else:
            elems.append(token)
            out = ('Data', token)
        token = yield out
    return elems






More information about the Python-ideas mailing list