Allow Context Managers to Support Suspended Execution
I do not think there is currently a good way for Context Managers to support suspended execution, as in await or yield. Both of these instructions cause the interpreter to leave the with block, yet no indication of this (temporary) exit or subsequent re-entrance is given to the context manager. If the intent of a Context Manager is to say "no matter how this block is entered or exited, the context will be correctly maintained", then this needs to be possible. I would propose magic methods __suspend__ and __resume__ as companions to the existing __enter__ and __exit__ methods (and their async variants). __suspend__, if present, would be called upon suspending execution on an await or yield statement, and __resume__, if present, would be called when execution is resumed. If __suspend__ or __resume__ are not present then nothing should be done, so that the behavior of existing context managers is preserved. Here is an example demonstrating the issue with await: https://gist.github.com/allemangD/bba8dc2d059310623f752ebf65bb6cdc and one with yield: https://gist.github.com/allemangD/f2534f16d3a0c642c2cdc02c544e854f The context manager used is clearly not thread-safe, and I'm not actually sure how to approach a thread-safe implementation with the proposed __suspend__ and __resume__ - but I don't believe that introducing these new methods would create any issues that aren't already present with __enter__ and __exit__. It's worth noting that the context manager used in those examples is, essentially, identical contextlib's redirect_stdout and decimal's localcontext managers. Any context manager such as these which modify global state or the behavior of global functions would benefit from this. It may also make sense to, for example, have the __suspend__ method on file objects flush buffers without closing the file, similar to their current __exit__ behavior, but I'm unsure what impact this would have on performance. It is important, though, that yield and await not use __enter__ or __exit__, as not all context-managers are reusable. I'm unsure what the best term would be to describe this type of context, as the documentation for contextlib already gives a different definition for "reentrant" - I would then call them "suspendable" contexts. It would make sense to have an @suspendable decorator, probably in contextlib, to indicate that a context manager can use __enter__ and __exit__ methods rather than __suspend__ and __resume__. All it would need to do is define __suspend__ to call __enter__() and __resume__ to call __exit__(None, None, None). It is also important, since __suspend__ and __resume__ would be called after a context is entered but before it is exited, that __suspend__ not accept any parameters and that __resume__ not use its return value. __suspend__ could not be triggered by an exception, only by a yield or await, and __resume__ could not have its return value named with as. Thanks, David
On 01/11/2018 02:52, David Allemang wrote:
I do not think there is currently a good way for Context Managers to support suspended execution, as in await or yield. Both of these instructions cause the interpreter to leave the with block, yet no indication of this (temporary) exit or subsequent re-entrance is given to the context manager. If the intent of a Context Manager is to say "no matter how this block is entered or exited, the context will be correctly maintained", then this needs to be possible.
I think you're going to have to justify this a bit more. From my point of view, yielding does not leave the with block in any meaningful sense. Indeed I'd be quite hacked off with a file context manager that was so inefficient as to close the file on yielding a line, only to have to re-open and seek when it got control back. -- Rhodri James *-* Kynesim Ltd
I'm very curious about the idea, but can't come up with any use cases based
just one your explanation. Maybe you could give some examples where this
would be useful? In particular, what are some cases that are really hard to
handle now and how would those cases be improved like this?
On Wed, Oct 31, 2018 at 10:53 PM David Allemang
I do not think there is currently a good way for Context Managers to support suspended execution, as in await or yield. Both of these instructions cause the interpreter to leave the with block, yet no indication of this (temporary) exit or subsequent re-entrance is given to the context manager. If the intent of a Context Manager is to say "no matter how this block is entered or exited, the context will be correctly maintained", then this needs to be possible.
I would propose magic methods __suspend__ and __resume__ as companions to the existing __enter__ and __exit__ methods (and their async variants). __suspend__, if present, would be called upon suspending execution on an await or yield statement, and __resume__, if present, would be called when execution is resumed. If __suspend__ or __resume__ are not present then nothing should be done, so that the behavior of existing context managers is preserved.
Here is an example demonstrating the issue with await: https://gist.github.com/allemangD/bba8dc2d059310623f752ebf65bb6cdc and one with yield: https://gist.github.com/allemangD/f2534f16d3a0c642c2cdc02c544e854f
The context manager used is clearly not thread-safe, and I'm not actually sure how to approach a thread-safe implementation with the proposed __suspend__ and __resume__ - but I don't believe that introducing these new methods would create any issues that aren't already present with __enter__ and __exit__.
It's worth noting that the context manager used in those examples is, essentially, identical contextlib's redirect_stdout and decimal's localcontext managers. Any context manager such as these which modify global state or the behavior of global functions would benefit from this. It may also make sense to, for example, have the __suspend__ method on file objects flush buffers without closing the file, similar to their current __exit__ behavior, but I'm unsure what impact this would have on performance.
It is important, though, that yield and await not use __enter__ or __exit__, as not all context-managers are reusable. I'm unsure what the best term would be to describe this type of context, as the documentation for contextlib already gives a different definition for "reentrant" - I would then call them "suspendable" contexts. It would make sense to have an @suspendable decorator, probably in contextlib, to indicate that a context manager can use __enter__ and __exit__ methods rather than __suspend__ and __resume__. All it would need to do is define __suspend__ to call __enter__() and __resume__ to call __exit__(None, None, None).
It is also important, since __suspend__ and __resume__ would be called after a context is entered but before it is exited, that __suspend__ not accept any parameters and that __resume__ not use its return value. __suspend__ could not be triggered by an exception, only by a yield or await, and __resume__ could not have its return value named with as.
Thanks,
David _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Check out the decimal example here:
https://www.python.org/dev/peps/pep-0568/ (PEP 568 is deferred, but PEP 567
is implemented in Python 3.7).
Those Contexts aren't context managers, but still there's some thought put
into swapping contexts out at the boundaries of generators.
On Wed, Oct 31, 2018 at 7:54 PM David Allemang
I do not think there is currently a good way for Context Managers to support suspended execution, as in await or yield. Both of these instructions cause the interpreter to leave the with block, yet no indication of this (temporary) exit or subsequent re-entrance is given to the context manager. If the intent of a Context Manager is to say "no matter how this block is entered or exited, the context will be correctly maintained", then this needs to be possible.
I would propose magic methods __suspend__ and __resume__ as companions to the existing __enter__ and __exit__ methods (and their async variants). __suspend__, if present, would be called upon suspending execution on an await or yield statement, and __resume__, if present, would be called when execution is resumed. If __suspend__ or __resume__ are not present then nothing should be done, so that the behavior of existing context managers is preserved.
Here is an example demonstrating the issue with await: https://gist.github.com/allemangD/bba8dc2d059310623f752ebf65bb6cdc and one with yield: https://gist.github.com/allemangD/f2534f16d3a0c642c2cdc02c544e854f
The context manager used is clearly not thread-safe, and I'm not actually sure how to approach a thread-safe implementation with the proposed __suspend__ and __resume__ - but I don't believe that introducing these new methods would create any issues that aren't already present with __enter__ and __exit__.
It's worth noting that the context manager used in those examples is, essentially, identical contextlib's redirect_stdout and decimal's localcontext managers. Any context manager such as these which modify global state or the behavior of global functions would benefit from this. It may also make sense to, for example, have the __suspend__ method on file objects flush buffers without closing the file, similar to their current __exit__ behavior, but I'm unsure what impact this would have on performance.
It is important, though, that yield and await not use __enter__ or __exit__, as not all context-managers are reusable. I'm unsure what the best term would be to describe this type of context, as the documentation for contextlib already gives a different definition for "reentrant" - I would then call them "suspendable" contexts. It would make sense to have an @suspendable decorator, probably in contextlib, to indicate that a context manager can use __enter__ and __exit__ methods rather than __suspend__ and __resume__. All it would need to do is define __suspend__ to call __enter__() and __resume__ to call __exit__(None, None, None).
It is also important, since __suspend__ and __resume__ would be called after a context is entered but before it is exited, that __suspend__ not accept any parameters and that __resume__ not use its return value. __suspend__ could not be triggered by an exception, only by a yield or await, and __resume__ could not have its return value named with as.
Thanks,
David _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
Yep, PEP 567 addresses this for coroutines, so David's first example
is covered; here's a link to the fixed version: [1]
The proposal to add __suspend__ and __resume__ is very similar to PEP
521 which was withdrawn. PEP 568 (which needs to be properly updated)
is the way to go if we want to address this issue for generators.
[1] https://gist.github.com/allemangD/bba8dc2d059310623f752ebf65bb6cdc#gistcomme...
Yury
On Thu, Nov 1, 2018 at 11:06 AM Guido van Rossum
Check out the decimal example here: https://www.python.org/dev/peps/pep-0568/ (PEP 568 is deferred, but PEP 567 is implemented in Python 3.7).
Those Contexts aren't context managers, but still there's some thought put into swapping contexts out at the boundaries of generators.
On Wed, Oct 31, 2018 at 7:54 PM David Allemang
wrote: I do not think there is currently a good way for Context Managers to support suspended execution, as in await or yield. Both of these instructions cause the interpreter to leave the with block, yet no indication of this (temporary) exit or subsequent re-entrance is given to the context manager. If the intent of a Context Manager is to say "no matter how this block is entered or exited, the context will be correctly maintained", then this needs to be possible.
I would propose magic methods __suspend__ and __resume__ as companions to the existing __enter__ and __exit__ methods (and their async variants). __suspend__, if present, would be called upon suspending execution on an await or yield statement, and __resume__, if present, would be called when execution is resumed. If __suspend__ or __resume__ are not present then nothing should be done, so that the behavior of existing context managers is preserved.
Here is an example demonstrating the issue with await: https://gist.github.com/allemangD/bba8dc2d059310623f752ebf65bb6cdc and one with yield: https://gist.github.com/allemangD/f2534f16d3a0c642c2cdc02c544e854f
The context manager used is clearly not thread-safe, and I'm not actually sure how to approach a thread-safe implementation with the proposed __suspend__ and __resume__ - but I don't believe that introducing these new methods would create any issues that aren't already present with __enter__ and __exit__.
It's worth noting that the context manager used in those examples is, essentially, identical contextlib's redirect_stdout and decimal's localcontext managers. Any context manager such as these which modify global state or the behavior of global functions would benefit from this. It may also make sense to, for example, have the __suspend__ method on file objects flush buffers without closing the file, similar to their current __exit__ behavior, but I'm unsure what impact this would have on performance.
It is important, though, that yield and await not use __enter__ or __exit__, as not all context-managers are reusable. I'm unsure what the best term would be to describe this type of context, as the documentation for contextlib already gives a different definition for "reentrant" - I would then call them "suspendable" contexts. It would make sense to have an @suspendable decorator, probably in contextlib, to indicate that a context manager can use __enter__ and __exit__ methods rather than __suspend__ and __resume__. All it would need to do is define __suspend__ to call __enter__() and __resume__ to call __exit__(None, None, None).
It is also important, since __suspend__ and __resume__ would be called after a context is entered but before it is exited, that __suspend__ not accept any parameters and that __resume__ not use its return value. __suspend__ could not be triggered by an exception, only by a yield or await, and __resume__ could not have its return value named with as.
Thanks,
David _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- Yury
Yes, so PEP 512 is exactly what I was suggesting. My apologies for not
finding it before sending this.
So, then, PEP 567 solves the issue for coroutines and PEP 568 would solve
it for generators as well?
On Thu, Nov 1, 2018, 11:40 AM Yury Selivanov Yep, PEP 567 addresses this for coroutines, so David's first example
is covered; here's a link to the fixed version: [1] The proposal to add __suspend__ and __resume__ is very similar to PEP
521 which was withdrawn. PEP 568 (which needs to be properly updated)
is the way to go if we want to address this issue for generators. [1]
https://gist.github.com/allemangD/bba8dc2d059310623f752ebf65bb6cdc#gistcomme... Yury
On Thu, Nov 1, 2018 at 11:06 AM Guido van Rossum Check out the decimal example here: https://www.python.org/dev/peps/pep-0568/ (PEP 568 is deferred, but PEP
567 is implemented in Python 3.7). Those Contexts aren't context managers, but still there's some thought put into swapping contexts out at the boundaries of generators. On Wed, Oct 31, 2018 at 7:54 PM David Allemang wrote: I do not think there is currently a good way for Context Managers to
support suspended execution, as in await or yield. Both of these
instructions cause the interpreter to leave the with block, yet no
indication of this (temporary) exit or subsequent re-entrance is given
to the context manager. If the intent of a Context Manager is to say
"no matter how this block is entered or exited, the context will be
correctly maintained", then this needs to be possible. I would propose magic methods __suspend__ and __resume__ as companions
to the existing __enter__ and __exit__ methods (and their async
variants). __suspend__, if present, would be called upon suspending
execution on an await or yield statement, and __resume__, if present,
would be called when execution is resumed. If __suspend__ or
__resume__ are not present then nothing should be done, so that the
behavior of existing context managers is preserved. Here is an example demonstrating the issue with await:
https://gist.github.com/allemangD/bba8dc2d059310623f752ebf65bb6cdc
and one with yield:
https://gist.github.com/allemangD/f2534f16d3a0c642c2cdc02c544e854f The context manager used is clearly not thread-safe, and I'm not
actually sure how to approach a thread-safe implementation with the
proposed __suspend__ and __resume__ - but I don't believe that
introducing these new methods would create any issues that aren't
already present with __enter__ and __exit__. It's worth noting that the context manager used in those examples is,
essentially, identical contextlib's redirect_stdout and decimal's
localcontext managers. Any context manager such as these which modify
global state or the behavior of global functions would benefit from
this. It may also make sense to, for example, have the __suspend__
method on file objects flush buffers without closing the file, similar
to their current __exit__ behavior, but I'm unsure what impact this
would have on performance. It is important, though, that yield and await not use __enter__ or
__exit__, as not all context-managers are reusable. I'm unsure what
the best term would be to describe this type of context, as the
documentation for contextlib already gives a different definition for
"reentrant" - I would then call them "suspendable" contexts. It would
make sense to have an @suspendable decorator, probably in contextlib,
to indicate that a context manager can use __enter__ and __exit__
methods rather than __suspend__ and __resume__. All it would need to
do is define __suspend__ to call __enter__() and __resume__ to call
__exit__(None, None, None). It is also important, since __suspend__ and __resume__ would be called
after a context is entered but before it is exited, that __suspend__
not accept any parameters and that __resume__ not use its return
value. __suspend__ could not be triggered by an exception, only by a
yield or await, and __resume__ could not have its return value named
with as. Thanks, David
_______________________________________________
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/ --
--Guido van Rossum (python.org/~guido)
_______________________________________________
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/ --
Yury
participants (5)
-
Calvin Spealman
-
David Allemang
-
Guido van Rossum
-
Rhodri James
-
Yury Selivanov