[Python-Dev] why _PyGen_Finalize(gen) propagates close() to _PyGen_yf() ?

[Martin Panter]

On 31 March 2017 at 05:22, Nathaniel Smith <njs at pobox.com> wrote:
>>> On Mon, Mar 20, 2017 at 11:30 AM, Oleg Nesterov <oleg at redhat.com> wrote:
>>> > [Aborting "yield" in a "for" loop leaves a sub-generator open, but aborting "yield from" cleans up the sub-generator]
>
> In any case the short answer to your original question is that PEP 342
> says that generator finalization calls the generator's close() method,
> which throws a GeneratorExit into the generator, and PEP 380 says that
> as a special case, when a GeneratorExit is thrown into a yield from,
> then this is propagated by calling .close() on the yielded-from
> iterator (if such a method exists) and then re-raised in the original
> generator.

I think the Python documentation could be improved regarding this.
When I wrote the documentation for coroutine methods
<https://docs.python.org/3/reference/datamodel.html#coroutine-objects>,
I included details about the "close" and "throw" methods delegating to
inner iterators. I thought I was going to propose similar updates to
the generator documentation
<https://docs.python.org/3/reference/expressions.html#generator-iterator-methods>,
but it seems I never got around to it. (In the mean time,
<https://docs.python.org/3/reference/expressions.html#asynchronous-generator-iterator-methods>
was added, to which this may also be relevant, but that is too
complicated for me.)

There is a parallel with another annoyance with Python generator
cleanup: <https://bugs.python.org/issue28629>. There are two ways you
can require generators to be used. With a simple generator, you can
partially iterate it and then throw it away without any special
cleaning up. But with more complex generators that "own" expensive
resources, it would be nice to produce a ResourceWarning I you forget
to clean them up.

With the "for / yield" case, the sub-generator is not cleaned up, so
if a resource-intensive sub-generator has to be cleaned up you have to
do it yourself. With "yield from", the cleanup is implicit and
unavoidable, which means you can't use it if you want keep the
sub-generator alive for later. But the automatic cleanup may be useful
in other cases.