I've now experienced an issue I believe that PEP 533 was intended to address: When an asynchronous context manager is created within an asynchronous generator, if the generator is not iterated fully, the context manager will not exit until the event loop cancels the task by raising a CancelledError, long after the context manager is assumed to be out of scope. Per PEP 525, I can call aclose coroutine method to cleanup the generator, but it requires the code iterating to be aware that that closing the generator is necessary. Any appetite for putting PEP 533 back on the table to address this kind of issue?
Hi Paul,
Per PEP 525, I can call aclose coroutine method to cleanup the generator, but it requires the code iterating to be aware that that closing the generator is necessary.
How about treating this as a bug for the specific use case that you mentioned, rather than a complete addition of " PEP 533 -- Deterministic cleanup for iterators". I am not certain why PEP 533 was deffered. I'd also suggest a documentation update if the details of aclose requirement is mentioned only in a PEP and not in the documentation. Thank you, Senthil
For what it's worth, this recent issue: https://bugs.python.org/issue42762 is somewhat related to the non-async aspect of 533. (generator func getting closed during __del__ and causing odd side-effects at seemingly random points in the execution flow) The example code provided in that issue is contrived, and involves catching unconditional exceptions and recursing, but following the code flow in that situation feels unreasonably hard, and deciding if it's a python bug or not is even harder. I'm not sure how easy an implementation of 533 would be that doesn't break a lot of existing code, but if one could be made, it might help make generator lifecycles more predictable. On Tue, Jan 5, 2021 at 1:08 PM Senthil Kumaran <senthil@uthcode.com> wrote:
Hi Paul,
Per PEP 525, I can call aclose coroutine method to cleanup the generator, but it requires the code iterating to be aware that that closing the generator is necessary.
How about treating this as a bug for the specific use case that you mentioned, rather than a complete addition of " PEP 533 -- Deterministic cleanup for iterators". I am not certain why PEP 533 was deffered.
I'd also suggest a documentation update if the details of aclose requirement is mentioned only in a PEP and not in the documentation.
Thank you, Senthil _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EPKMYH7D... Code of Conduct: http://python.org/psf/codeofconduct/
I, too, think I ran into basically the same situation as Paul: An outer async generator calls an inner async generator. The outer doesn't fully consume the inner. The inner uses an async context manager. The inner CM isn't exited when expected and resources aren't released when required/expected. FWIW, the analysis of PEP 533 seems spot on -- for synchronous code, while still possible, it's harder to trigger thanks to the GC tricks. For asynchronous code, it easily (and subtly) results in resources not being released when expected. Or perhaps never at all? IIUC, a long-lived server might just accumulate unfinished async generators indefinitely, or at least until some OS resource was exhausted. In my particular case, I have thousands of network requests to make, each of which is potentially large and/or slow to complete in its entirety, so I'm taking extra care to break early to minimize resource usage to both client and servers. I spent about an entire day trying to figure why the synchronous version of my code Just Worked, while the asynchronous version simply refused to close things when I was done with them. It was only after I started looking at third party libraries and stumbled upon a reference to PEP 533 and a detailed blog post from 5 years ago <https://async-generator.readthedocs.io/en/latest/reference.html#garbage-collection-hooks> that I had a clue what was going on. My point being, the feature PEP 533 describes would have made things Just Work in line with my expectations. On Tue, Jan 5, 2021 at 8:35 AM Stestagg <stestagg@gmail.com> wrote:
For what it's worth, this recent issue: https://bugs.python.org/issue42762 is somewhat related to the non-async aspect of 533. (generator func getting closed during __del__ and causing odd side-effects at seemingly random points in the execution flow)
The example code provided in that issue is contrived, and involves catching unconditional exceptions and recursing, but following the code flow in that situation feels unreasonably hard, and deciding if it's a python bug or not is even harder.
I'm not sure how easy an implementation of 533 would be that doesn't break a lot of existing code, but if one could be made, it might help make generator lifecycles more predictable.
On Tue, Jan 5, 2021 at 1:08 PM Senthil Kumaran <senthil@uthcode.com> wrote:
Hi Paul,
Per PEP 525, I can call aclose coroutine method to cleanup the generator, but it requires the code iterating to be aware that that closing the generator is necessary.
How about treating this as a bug for the specific use case that you mentioned, rather than a complete addition of " PEP 533 -- Deterministic cleanup for iterators". I am not certain why PEP 533 was deffered.
I'd also suggest a documentation update if the details of aclose requirement is mentioned only in a PEP and not in the documentation.
Thank you, Senthil _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EPKMYH7D... Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VB64QJAY... Code of Conduct: http://python.org/psf/codeofconduct/
For what it's worth, I've increasingly considered creating a context in a generator (sync or async) to be an antipattern. To this end, in my case, I moved creation of the context outside of the generator. This resulted in a more explicit contract, which actually simplified reasoning about its behavior. On Sun, 2021-02-07 at 16:18 -0800, Richard Levasseur wrote:
I, too, think I ran into basically the same situation as Paul: An outer async generator calls an inner async generator. The outer doesn't fully consume the inner. The inner uses an async context manager. The inner CM isn't exited when expected and resources aren't released when required/expected.
FWIW, the analysis of PEP 533 seems spot on -- for synchronous code, while still possible, it's harder to trigger thanks to the GC tricks. For asynchronous code, it easily (and subtly) results in resources not being released when expected. Or perhaps never at all? IIUC, a long-lived server might just accumulate unfinished async generators indefinitely, or at least until some OS resource was exhausted.
In my particular case, I have thousands of network requests to make, each of which is potentially large and/or slow to complete in its entirety, so I'm taking extra care to break early to minimize resource usage to both client and servers. I spent about an entire day trying to figure why the synchronous version of my code Just Worked, while the asynchronous version simply refused to close things when I was done with them. It was only after I started looking at third party libraries and stumbled upon a reference to PEP 533 and a detailed blog post from 5 years ago that I had a clue what was going on. My point being, the feature PEP 533 describes would have made things Just Work in line with my expectations.
On Tue, Jan 5, 2021 at 8:35 AM Stestagg <stestagg@gmail.com> wrote:
For what it's worth, this recent issue: https://bugs.python.org/issue42762 is somewhat related to the non- async aspect of 533. (generator func getting closed during __del__ and causing odd side-effects at seemingly random points in the execution flow)
The example code provided in that issue is contrived, and involves catching unconditional exceptions and recursing, but following the code flow in that situation feels unreasonably hard, and deciding if it's a python bug or not is even harder.
I'm not sure how easy an implementation of 533 would be that doesn't break a lot of existing code, but if one could be made, it might help make generator lifecycles more predictable.
On Tue, Jan 5, 2021 at 1:08 PM Senthil Kumaran <senthil@uthcode.com> wrote:
Hi Paul,
Per PEP 525, I can call aclose coroutine method to cleanup the generator, but it requires the code iterating to be aware that that closing the generator is necessary.
How about treating this as a bug for the specific use case that you mentioned, rather than a complete addition of " PEP 533 -- Deterministic cleanup for iterators". I am not certain why PEP 533 was deffered.
I'd also suggest a documentation update if the details of aclose requirement is mentioned only in a PEP and not in the documentation.
Thank you, Senthil _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/EPKMYH7D...
Code of Conduct: http://python.org/psf/codeofconduct/
Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/VB64QJAY...
Code of Conduct: http://python.org/psf/codeofconduct/
Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/D7CO5ZMI... Code of Conduct: http://python.org/psf/codeofconduct/
participants (4)
-
Paul Bryan -
Richard Levasseur -
Senthil Kumaran -
Stestagg