why _PyGen_Finalize(gen) propagates close() to _PyGen_yf() ?

Hello, Let me first clarify, I do not claim this is a bug, I am trying to learn python and now I trying to understand yield-from. This simple test-case g = (x for x in range(10)) def fg(): for x in g: yield x print(next(fg())) print(next(g)) works as expected and prints: 0 1 However, if I change fg() to use yield-from g = (x for x in range(10)) def fg(): yield from g print(next(fg())) print(next(g)) then next(g) raises StopIteration: 0 Traceback (most recent call last): File "/tmp/T.py", line 10, in <module> print(next(g)) StopIteration because g.close() is called by destructor of the object returned by fg(). To me this looks strange and confusing. I tried to google, and found https://docs.python.org/3/whatsnew/3.3.html#pep-380 but it doesn't document this behaviour. I understand that yield-from should propagate .close(), but why _PyGen_Finalize() should send close() to the gi_yieldfrom object? I applied the patch below just to verify that I actually understand what's going on, and with this patch the 2nd test-case works as I'd expect. But since I am very new to python I'd suspect that the code is fine and I simply do not understand why it works this way. So. could someone please explain the rationale behind this behaviour? And probably update the docs should be updated? Thanks in advance. Oleg. --- diff --git a/Objects/genobject.c b/Objects/genobject.c index 24a1da6..d5152eb 100644 --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -6,6 +6,7 @@ #include "opcode.h" static PyObject *gen_close(PyGenObject *, PyObject *); +static PyObject *do_gen_close(PyGenObject *, PyObject *); static PyObject *async_gen_asend_new(PyAsyncGenObject *, PyObject *); static PyObject *async_gen_athrow_new(PyAsyncGenObject *, PyObject *); @@ -71,7 +72,7 @@ _PyGen_Finalize(PyObject *self) } } else { - res = gen_close(gen, NULL); + res = do_gen_close(gen, NULL); } if (res == NULL) { @@ -373,10 +374,9 @@ _PyGen_yf(PyGenObject *gen) } static PyObject * -gen_close(PyGenObject *gen, PyObject *args) +do_gen_close(PyGenObject *gen, PyObject *yf) { PyObject *retval; - PyObject *yf = _PyGen_yf(gen); int err = 0; if (yf) { @@ -407,6 +407,11 @@ gen_close(PyGenObject *gen, PyObject *args) return NULL; } +static PyObject * +gen_close(PyGenObject *gen, PyObject *args) +{ + return do_gen_close(gen, _PyGen_yf(gen)); +} PyDoc_STRVAR(throw_doc, "throw(typ[,val[,tb]]) -> raise exception in generator,\n\

On 03/28, Eric Snow wrote:
and this looks a bit strange. My question is simple, the implementation looks clear and straightforward, I am a bit surprised none of cpython devs bothered to reply.
you may want to take this over to the core-mentorship@python.org list.
Well, I'm afraid to contact this closed and not-for-mortals list, not sure this very basic question should go there ;) perhaps you are already a member, feel free to forward. I downloaded micropython and it doesn't propagate .close() in this case. but it seem to differ very much from cpython, not sure this matters at all. Thanks, Oleg.

On Thu, Mar 30, 2017 at 11:05 AM, Oleg Nesterov <oleg@redhat.com> wrote:
core-mentorship is intended as a friendly place for folks who are starting to study CPython internals. I'm not sure where you got the impression that it's not-for-mortals but I suspect the people running it would like to know so they can fix it :-). In any case the short answer to your original question is that PEP 342 says that generator finalization calls the generator's close() method, which throws a GeneratorExit into the generator, and PEP 380 says that as a special case, when a GeneratorExit is thrown into a yield from, then this is propagated by calling .close() on the yielded-from iterator (if such a method exists) and then re-raised in the original generator. -n -- Nathaniel J. Smith -- https://vorpus.org

On 31 March 2017 at 05:22, Nathaniel Smith <njs@pobox.com> wrote:
I think the Python documentation could be improved regarding this. When I wrote the documentation for coroutine methods <https://docs.python.org/3/reference/datamodel.html#coroutine-objects>, I included details about the "close" and "throw" methods delegating to inner iterators. I thought I was going to propose similar updates to the generator documentation <https://docs.python.org/3/reference/expressions.html#generator-iterator-meth...>, but it seems I never got around to it. (In the mean time, <https://docs.python.org/3/reference/expressions.html#asynchronous-generator-...> was added, to which this may also be relevant, but that is too complicated for me.) There is a parallel with another annoyance with Python generator cleanup: <https://bugs.python.org/issue28629>. There are two ways you can require generators to be used. With a simple generator, you can partially iterate it and then throw it away without any special cleaning up. But with more complex generators that "own" expensive resources, it would be nice to produce a ResourceWarning I you forget to clean them up. With the "for / yield" case, the sub-generator is not cleaned up, so if a resource-intensive sub-generator has to be cleaned up you have to do it yourself. With "yield from", the cleanup is implicit and unavoidable, which means you can't use it if you want keep the sub-generator alive for later. But the automatic cleanup may be useful in other cases.

Nathaniel Smith writes:
I imagine Oleg got that impression because he tried to access the archives, which are closed to non-members. I don't recall the rationale for that but there was one, I suspect that won't change. IMO, the listinfo https://mail.python.org/mailman/listinfo/core-mentorship makes it clear that developers new to Python core code (whether they are new or advanced developers in other code bases) are welcome. If it's something else, we would *definitely* like to know. Steve

On 3 April 2017 at 05:23, Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
It relates to these two points in the list's guidelines for participation: ========= * Statements made by core developers can be quoted outside of the list. * Statements made by others can not be quoted outside the list without explicit permission. ========= as well as this stated goal: ========= A major goal of this group is to help new contributors feel more confident about participating in the results focused public environments of the bug tracker, python-dev and python-ideas. ========= Lots of folks aren't comfortable with having their newbie questions being archived permanently in reach of online search engines, so rather than saying "You shouldn't be bothered by that", we instead offer them a place to ask questions that is at least somewhat private. There are folks at the other end of the spectrum that don't trust any communications channel with deliberately closed archives ("What are they hiding?"), but that's OK - different services are provided for different audiences. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 03/28, Eric Snow wrote:
and this looks a bit strange. My question is simple, the implementation looks clear and straightforward, I am a bit surprised none of cpython devs bothered to reply.
you may want to take this over to the core-mentorship@python.org list.
Well, I'm afraid to contact this closed and not-for-mortals list, not sure this very basic question should go there ;) perhaps you are already a member, feel free to forward. I downloaded micropython and it doesn't propagate .close() in this case. but it seem to differ very much from cpython, not sure this matters at all. Thanks, Oleg.

On Thu, Mar 30, 2017 at 11:05 AM, Oleg Nesterov <oleg@redhat.com> wrote:
core-mentorship is intended as a friendly place for folks who are starting to study CPython internals. I'm not sure where you got the impression that it's not-for-mortals but I suspect the people running it would like to know so they can fix it :-). In any case the short answer to your original question is that PEP 342 says that generator finalization calls the generator's close() method, which throws a GeneratorExit into the generator, and PEP 380 says that as a special case, when a GeneratorExit is thrown into a yield from, then this is propagated by calling .close() on the yielded-from iterator (if such a method exists) and then re-raised in the original generator. -n -- Nathaniel J. Smith -- https://vorpus.org

On 31 March 2017 at 05:22, Nathaniel Smith <njs@pobox.com> wrote:
I think the Python documentation could be improved regarding this. When I wrote the documentation for coroutine methods <https://docs.python.org/3/reference/datamodel.html#coroutine-objects>, I included details about the "close" and "throw" methods delegating to inner iterators. I thought I was going to propose similar updates to the generator documentation <https://docs.python.org/3/reference/expressions.html#generator-iterator-meth...>, but it seems I never got around to it. (In the mean time, <https://docs.python.org/3/reference/expressions.html#asynchronous-generator-...> was added, to which this may also be relevant, but that is too complicated for me.) There is a parallel with another annoyance with Python generator cleanup: <https://bugs.python.org/issue28629>. There are two ways you can require generators to be used. With a simple generator, you can partially iterate it and then throw it away without any special cleaning up. But with more complex generators that "own" expensive resources, it would be nice to produce a ResourceWarning I you forget to clean them up. With the "for / yield" case, the sub-generator is not cleaned up, so if a resource-intensive sub-generator has to be cleaned up you have to do it yourself. With "yield from", the cleanup is implicit and unavoidable, which means you can't use it if you want keep the sub-generator alive for later. But the automatic cleanup may be useful in other cases.

Nathaniel Smith writes:
I imagine Oleg got that impression because he tried to access the archives, which are closed to non-members. I don't recall the rationale for that but there was one, I suspect that won't change. IMO, the listinfo https://mail.python.org/mailman/listinfo/core-mentorship makes it clear that developers new to Python core code (whether they are new or advanced developers in other code bases) are welcome. If it's something else, we would *definitely* like to know. Steve

On 3 April 2017 at 05:23, Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
It relates to these two points in the list's guidelines for participation: ========= * Statements made by core developers can be quoted outside of the list. * Statements made by others can not be quoted outside the list without explicit permission. ========= as well as this stated goal: ========= A major goal of this group is to help new contributors feel more confident about participating in the results focused public environments of the bug tracker, python-dev and python-ideas. ========= Lots of folks aren't comfortable with having their newbie questions being archived permanently in reach of online search engines, so rather than saying "You shouldn't be bothered by that", we instead offer them a place to ask questions that is at least somewhat private. There are folks at the other end of the spectrum that don't trust any communications channel with deliberately closed archives ("What are they hiding?"), but that's OK - different services are provided for different audiences. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (6)
-
Eric Snow
-
Martin Panter
-
Nathaniel Smith
-
Nick Coghlan
-
Oleg Nesterov
-
Stephen J. Turnbull