Coroutines: unexpected behaviour

[Thomas Jollans]

On 06/16/2010 06:35 PM, Steven D'Aprano wrote:
> On Wed, 16 Jun 2010 05:03:13 -0700, Jérôme Mainka wrote:
> 
>> Hello,
>>
>> I try to experiment with coroutines and I don't understand why this
>> snippet doesn't work as expected... In python 2.5 and python 2.6 I get
>> the following output:
>>
>> 0
>> Exception exceptions.TypeError: "'NoneType' object is not callable" in
>> <generator object at 0x7e43f8> ignored

Heureka! I think I've found the answer.

Have a look at this:

###
from itertools import izip

def gen():
    print globals()
    try:
        while True:
            yield 'things'
    except GeneratorExit:
        print globals()

if __name__ == "__main__":
    g = gen()
    g.next()
###
% python genexit.py
{'izip': <type 'itertools.izip'>, 'g': <generator object at
0x7f76c7340878>, '__builtins__': <module '__builtin__' (built-in)>,
'__file__': 'genexit.py', 'gen': <function gen at 0x7f76c7327848>,
'__name__': '__main__', '__doc__': None}
{'izip': None, 'g': None, '__builtins__': <module '__builtin__'
(built-in)>, '__file__': 'genexit.py', 'gen': <function gen at
0x7f76c7327848>, '__name__': '__main__', '__doc__': None}
###

Note that izip (the thing I imported, could just as well have been
pprint) is None the second time I print globals(). why?

This second print happens when GeneratorExit is raised. Just like all
occurences of dump.send(..) in the OP's code are underneath
"except GeneratorExit:"
When does GeneratorExit happen? Simple: When the generator is destroyed.
And when is it destroyed? *looks at the end of the file*
Since it's a global variable, it's destroyed when the module is
destroyed. Python sees the end of the script, and del's the module,
which means all the items in the module's __dict__ are del'd one-by-one
and, apparently, to avoide causing havoc and mayhem, replaced by None.
Fair enough.

So why did the "solutions" we found work?

(1) changing the variable name:
I expect Python del's the globals in some particular order, and this
order is probably influenced by the key in the globals() dict. One key
happens to be deleted before pprint (then it works, pprint is still
there), another one happens to come after pprint. For all I know, it
might not even be deterministic.

(2) moving pprint into the dump() function
As long as there's a reference to the dump generator around, there will
be a reference to pprint around. sort() has a local reference to
target == dump(), so no problem there.

(3) wrapping the code in a function
As the generator is now a local object, it gets destroyed with the
locals, when the function exits. The module gets destroyed afterwards.


What should you do to fix it then?

If you really want to keep the except GeneratorExit: approach, make sure
you exit it manually. Though really, you should do something like
p.send(None) at the end, and check for that in the generator: recieving
None would mean "we're done here, do post processing!"

-- Thomas