Adding context manager interface to contextvars.Token
It seems that the following idea has not been raised here before. If it has been, I'd be grateful for a pointer to the relevant discussion. The idea is: how about adding context manager interface (__enter__ and __exit__ methods) to the contextvars.Token type? This would allow code like this: with var.set(1): # var.get() == 1 that would be equivalent to the following more verbose snippet (taken from PEP 567): token = var.set(1) try: # var.get() == 1 finally: var.reset(token) I attach a very rough proof-of-concept implementation of the idea. The proper way to implement this proposal would be of course to modify the _contextvars C-extension. Here is my motivation for this proposal: The contextvars module is promoted as a replacement for thread-local storage for asynchronous programming, but in fact it seems to me that ContextVars implement thread-safe and async-safe dynamic scoping [1] in Python. One of the uses of dynamic scoping is as an alternative to function parameters for library configuration [2]. While dynamic scoping by default (as in Emacs Lisp) can be dangerous, I believe that explicit dynamic scoping could be useful in the Python world as a means to avoid the "parameter hell" of some libraries, perhaps most infamously demonstrated by matplotlib. With the updated ContextVars, a hypothetical plotting library could be used like this: with plotting.line_thickness.set(2), plotting.line_style('dashed'): plotting.plot(x, y) As explained in [2], the advantages of this approach compared to argument passing is that other functions that internally use plotting.plot do not have to expose all of its configuration options by themselves as parameters. Also, with the same mechanism it is possible to set a parameter for a single invocation of plotting.plot, or a default value for a whole script. [1] https://en.wikipedia.org/wiki/Scope_(computer_science)#Dynamic_scoping [2] https://www.gnu.org/software/emacs/emacs-paper.html#SEC18
Hi Christoph, Adding context manager protocol support to contextvars.Token was considered when PEP 567 was discussed. There wasn't a strong argument against that; however we decided not to immediately add it because context variables is a relatively low-level API. In you case, you can simply wrap a ContextVar in a context manager: _setting = contextvars.ContextVar('setting') @contextlib.contextmanager def setting(value): tok = _setting.set(value) try: yield finally: _setting.reset(tok) and later: with setting(something): ... Yury On Wed, Jun 5, 2019 at 11:19 AM Christoph Groth <christoph@grothesque.org> wrote:
-- Yury
Hi Yury, thanks for the quick reply. Yury Selivanov wrote:
I'm aware of this possibility, however it is not suitable for the use case that I have in mind (configuration variables), because it requires too much code for each variable. Of course one could wrap your example inside a factory function, but that's unwieldy as well, since it requires keeping track of one context manager for each context variable: def factory(name): _setting = contextvars.ContextVar(name) @contextlib.contextmanager def setting(value): tok = _setting.set(value) try: yield finally: _setting.reset(tok) return _setting, setting One possibility to imlement this cleanly and efficiently without modifying the standard library module would be to derive classes from ContextVar and from Token, but this is not possible. So it seems to me that if the use of context variables as explicit dynamically bound variables for library configuration is considered interesting, one should think about adding __enter__ and __exit__ to contextvars.Token. By the way, contexts being implemented as immutable dictionaries implies one copy of a dictionary per call to the set method. That in turn means that using hundreds of context variables (a large library could easily have that many configuration variables) might not be a good idea. Is this correct? A final, unrelated comment: I find the use of the word "context" in the standard library for both context managers and context variables (and related to them "contexts") needlessly confusing. I suggest clearly explaining their independence at the top of the documentation of either module.
On Wed, Jun 5, 2019 at 6:22 PM Christoph Groth <christoph@grothesque.org> wrote: [..]
I suggest you to open an issue on bugs.python.org to implement support for context manager protocol for contextvars.Token. I'm not opposed to the idea. Keep in mind that Python 3.8 is already in a feature freeze mode, so the earliest we can get this is Python 3.9.
The contextvars module uses a special dictionary implementation (read more on that in PEP 567/550) with its ".set()" operation only slightly slower than updating a standard Python dict. Setting/resetting N context variables is about 1.5x slower than mutating a Python dict N times. In short, it's a fast operation by Python standards.
Pull requests to improve the docs are always welcome! Yury
I had a similar recent need, with a bit more on top of it, and solved it with this slightly insane library. (Alas, I haven't figured out a good way to make it act as a true subtype of UnderlyingType yet) import contextvars from typing import Any, Generic, TypeVar UnderlyingType = TypeVar('UnderlyingType') class DuckTypeContextVar(Generic[UnderlyingType]): """A DuckTypeContextVar takes contextvars.ContextVar to the next level: It duck-type emulates the underlying variable. This means you can do something like sys.stdout = DuckTypeContextVar('stdout', sys.stdout) token = sys.stdout.setLocalValue(myFile) .... sys.stdout.resetLocalValue(token) """ def __init__(self, name: str, value: UnderlyingType) -> None: self._value = contextvars.ContextVar(name, default=value) def getLocalValue(self) -> UnderlyingType: return self._value.get() def setLocalValue(self, value: UnderlyingType) -> contextvars.Token: return self._value.set(value) def resetLocalValue(self, token: contextvars.Token) -> None: self._value.reset(token) def __getattr__(self, attr: str) -> Any: return getattr(self._value, attr) On Wed, Jun 5, 2019 at 6:14 PM Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Yonatan Zunger wrote:
To me, your trick seems to address a different problem. You want context variables to be transparent and behave themselves as the data that they hold (in C++ terms: you want reference semantics instead of pointer semantics). This trick might be necessary at some times (like your example of replacing sys.stdout with a context var), but I consider it dangerous for general usage.
On Thu, Jun 6, 2019 at 4:15 AM Christoph Groth <christoph@grothesque.org> wrote:
Oh, absolutely. There's a reason I wasn't proposing that this be added to any standard library. :) That said, it (and other things) have made me think that it would be very nice to have a better way of expressing duck-type inheritance in the typing system that doesn't rely on the superclass declaring itself in particular ways, but that's an entirely separate issue and one I haven't thought through in any detail yet.
Yury Selivanov wrote:
I'll do that. Le me just raise one more aspect: for the usage of context vars as configuration variables (and likely for other uses as well), it would be useful if there was some way to validate new values when they are set. This would provide immediate feedback to the user and would avoid having to check them at every use. Did you consider anything like that? For example, ContextVar could accept an optional keyword arg 'validate' that must be a function that is then called for each new value, and somehow reports problems (by return value or by raising an exception). (A somewhat related issue would be support for ContextVars in the typing module.)
Thanks.
Sure, I just wanted to double check whether I'm overseeing anything here or whether both kinds of contexts are indeed completely independent concepts.
On Thu, Jun 6, 2019 at 8:04 AM Christoph Groth <christoph@grothesque.org> wrote:
For that you should definitely use a wrapper. Let's use composition instead of complicating the existing APIs.
(A somewhat related issue would be support for ContextVars in the typing module.)
contextvars should already support this syntax: c: ContextVar[int] = ContextVar('c') Now it's up to mypy et al to support this if they don't already (I haven't checked that.) [..]
They aren't completely independent, as it's quite useful to store context manager's state in a context variable (decimal context is a great example of that). OTOH I can totally understand why people can be confused about this. I would love to improve the docs to lessen the confusion where possible, but we're going to keep using the term for both "context managers" and "context variables" as it's too late to change that. -- Yury
Yury Selivanov wrote:
Agree in principle, but how would that work? If it was possible to derive a Python class from ContextVar, one could add the checking easily and transparently. But unfortunately that's not possible (perhaps it could be fixed?). The other possibility I see would be wrapping ContextVar inside a class (like in the example module that I attached to the first post in this thread), but that's not a good solution. It requires replicating the complete API, and is hence slow and not forward-compatible.
On Thu, Jun 6, 2019 at 5:20 PM Christoph Groth <christoph@grothesque.org> wrote: [..]
Wrapping isn't much slower than subclassing and calling super(). Indeed, you'll need to write some code that forwards calls to lower-level contextvars APIs, but that's fine. Enhancing contextvars with validators to satisfy your very specific use case would make contextvars slightly slower for all users that don't need that. I'm afraid that wrappers (composition) is the only way here. Yury
Yury Selivanov wrote:
What is actually the reason that Py_TPFLAGS_BASETYPE is not set for ContextVar (and many other built-in types)? The performance penality cannot be significant since both tuple and list are subclassable. Does it complicate the implementation? The requirements [1] do not look bad. Am I missing anything obvious here? (The extension types that I wrote are also not subclassable, but I just followed the default here and so far no one complained.) [1] https://www.python.org/dev/peps/pep-0253/#preparing-a-type-for-subtyping
Hi Christoph, Adding context manager protocol support to contextvars.Token was considered when PEP 567 was discussed. There wasn't a strong argument against that; however we decided not to immediately add it because context variables is a relatively low-level API. In you case, you can simply wrap a ContextVar in a context manager: _setting = contextvars.ContextVar('setting') @contextlib.contextmanager def setting(value): tok = _setting.set(value) try: yield finally: _setting.reset(tok) and later: with setting(something): ... Yury On Wed, Jun 5, 2019 at 11:19 AM Christoph Groth <christoph@grothesque.org> wrote:
-- Yury
Hi Yury, thanks for the quick reply. Yury Selivanov wrote:
I'm aware of this possibility, however it is not suitable for the use case that I have in mind (configuration variables), because it requires too much code for each variable. Of course one could wrap your example inside a factory function, but that's unwieldy as well, since it requires keeping track of one context manager for each context variable: def factory(name): _setting = contextvars.ContextVar(name) @contextlib.contextmanager def setting(value): tok = _setting.set(value) try: yield finally: _setting.reset(tok) return _setting, setting One possibility to imlement this cleanly and efficiently without modifying the standard library module would be to derive classes from ContextVar and from Token, but this is not possible. So it seems to me that if the use of context variables as explicit dynamically bound variables for library configuration is considered interesting, one should think about adding __enter__ and __exit__ to contextvars.Token. By the way, contexts being implemented as immutable dictionaries implies one copy of a dictionary per call to the set method. That in turn means that using hundreds of context variables (a large library could easily have that many configuration variables) might not be a good idea. Is this correct? A final, unrelated comment: I find the use of the word "context" in the standard library for both context managers and context variables (and related to them "contexts") needlessly confusing. I suggest clearly explaining their independence at the top of the documentation of either module.
On Wed, Jun 5, 2019 at 6:22 PM Christoph Groth <christoph@grothesque.org> wrote: [..]
I suggest you to open an issue on bugs.python.org to implement support for context manager protocol for contextvars.Token. I'm not opposed to the idea. Keep in mind that Python 3.8 is already in a feature freeze mode, so the earliest we can get this is Python 3.9.
The contextvars module uses a special dictionary implementation (read more on that in PEP 567/550) with its ".set()" operation only slightly slower than updating a standard Python dict. Setting/resetting N context variables is about 1.5x slower than mutating a Python dict N times. In short, it's a fast operation by Python standards.
Pull requests to improve the docs are always welcome! Yury
I had a similar recent need, with a bit more on top of it, and solved it with this slightly insane library. (Alas, I haven't figured out a good way to make it act as a true subtype of UnderlyingType yet) import contextvars from typing import Any, Generic, TypeVar UnderlyingType = TypeVar('UnderlyingType') class DuckTypeContextVar(Generic[UnderlyingType]): """A DuckTypeContextVar takes contextvars.ContextVar to the next level: It duck-type emulates the underlying variable. This means you can do something like sys.stdout = DuckTypeContextVar('stdout', sys.stdout) token = sys.stdout.setLocalValue(myFile) .... sys.stdout.resetLocalValue(token) """ def __init__(self, name: str, value: UnderlyingType) -> None: self._value = contextvars.ContextVar(name, default=value) def getLocalValue(self) -> UnderlyingType: return self._value.get() def setLocalValue(self, value: UnderlyingType) -> contextvars.Token: return self._value.set(value) def resetLocalValue(self, token: contextvars.Token) -> None: self._value.reset(token) def __getattr__(self, attr: str) -> Any: return getattr(self._value, attr) On Wed, Jun 5, 2019 at 6:14 PM Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Yonatan Zunger wrote:
To me, your trick seems to address a different problem. You want context variables to be transparent and behave themselves as the data that they hold (in C++ terms: you want reference semantics instead of pointer semantics). This trick might be necessary at some times (like your example of replacing sys.stdout with a context var), but I consider it dangerous for general usage.
On Thu, Jun 6, 2019 at 4:15 AM Christoph Groth <christoph@grothesque.org> wrote:
Oh, absolutely. There's a reason I wasn't proposing that this be added to any standard library. :) That said, it (and other things) have made me think that it would be very nice to have a better way of expressing duck-type inheritance in the typing system that doesn't rely on the superclass declaring itself in particular ways, but that's an entirely separate issue and one I haven't thought through in any detail yet.
Yury Selivanov wrote:
I'll do that. Le me just raise one more aspect: for the usage of context vars as configuration variables (and likely for other uses as well), it would be useful if there was some way to validate new values when they are set. This would provide immediate feedback to the user and would avoid having to check them at every use. Did you consider anything like that? For example, ContextVar could accept an optional keyword arg 'validate' that must be a function that is then called for each new value, and somehow reports problems (by return value or by raising an exception). (A somewhat related issue would be support for ContextVars in the typing module.)
Thanks.
Sure, I just wanted to double check whether I'm overseeing anything here or whether both kinds of contexts are indeed completely independent concepts.
On Thu, Jun 6, 2019 at 8:04 AM Christoph Groth <christoph@grothesque.org> wrote:
For that you should definitely use a wrapper. Let's use composition instead of complicating the existing APIs.
(A somewhat related issue would be support for ContextVars in the typing module.)
contextvars should already support this syntax: c: ContextVar[int] = ContextVar('c') Now it's up to mypy et al to support this if they don't already (I haven't checked that.) [..]
They aren't completely independent, as it's quite useful to store context manager's state in a context variable (decimal context is a great example of that). OTOH I can totally understand why people can be confused about this. I would love to improve the docs to lessen the confusion where possible, but we're going to keep using the term for both "context managers" and "context variables" as it's too late to change that. -- Yury
Yury Selivanov wrote:
Agree in principle, but how would that work? If it was possible to derive a Python class from ContextVar, one could add the checking easily and transparently. But unfortunately that's not possible (perhaps it could be fixed?). The other possibility I see would be wrapping ContextVar inside a class (like in the example module that I attached to the first post in this thread), but that's not a good solution. It requires replicating the complete API, and is hence slow and not forward-compatible.
On Thu, Jun 6, 2019 at 5:20 PM Christoph Groth <christoph@grothesque.org> wrote: [..]
Wrapping isn't much slower than subclassing and calling super(). Indeed, you'll need to write some code that forwards calls to lower-level contextvars APIs, but that's fine. Enhancing contextvars with validators to satisfy your very specific use case would make contextvars slightly slower for all users that don't need that. I'm afraid that wrappers (composition) is the only way here. Yury
Yury Selivanov wrote:
What is actually the reason that Py_TPFLAGS_BASETYPE is not set for ContextVar (and many other built-in types)? The performance penality cannot be significant since both tuple and list are subclassable. Does it complicate the implementation? The requirements [1] do not look bad. Am I missing anything obvious here? (The extension types that I wrote are also not subclassable, but I just followed the default here and so far no one complained.) [1] https://www.python.org/dev/peps/pep-0253/#preparing-a-type-for-subtyping
participants (3)
-
Christoph Groth -
Yonatan Zunger -
Yury Selivanov