Adding context manager interface to contextvars.Token
It seems that the following idea has not been raised here before. If it has been, I'd be grateful for a pointer to the relevant discussion. The idea is: how about adding context manager interface (__enter__ and __exit__ methods) to the contextvars.Token type? This would allow code like this: with var.set(1): # var.get() == 1 that would be equivalent to the following more verbose snippet (taken from PEP 567): token = var.set(1) try: # var.get() == 1 finally: var.reset(token) I attach a very rough proof-of-concept implementation of the idea. The proper way to implement this proposal would be of course to modify the _contextvars C-extension. Here is my motivation for this proposal: The contextvars module is promoted as a replacement for thread-local storage for asynchronous programming, but in fact it seems to me that ContextVars implement thread-safe and async-safe dynamic scoping [1] in Python. One of the uses of dynamic scoping is as an alternative to function parameters for library configuration [2]. While dynamic scoping by default (as in Emacs Lisp) can be dangerous, I believe that explicit dynamic scoping could be useful in the Python world as a means to avoid the "parameter hell" of some libraries, perhaps most infamously demonstrated by matplotlib. With the updated ContextVars, a hypothetical plotting library could be used like this: with plotting.line_thickness.set(2), plotting.line_style('dashed'): plotting.plot(x, y) As explained in [2], the advantages of this approach compared to argument passing is that other functions that internally use plotting.plot do not have to expose all of its configuration options by themselves as parameters. Also, with the same mechanism it is possible to set a parameter for a single invocation of plotting.plot, or a default value for a whole script. [1] https://en.wikipedia.org/wiki/Scope_(computer_science)#Dynamic_scoping [2] https://www.gnu.org/software/emacs/emacs-paper.html#SEC18
Hi Christoph,
Adding context manager protocol support to contextvars.Token was
considered when PEP 567 was discussed. There wasn't a strong argument
against that; however we decided not to immediately add it because
context variables is a relatively low-level API. In you case, you can
simply wrap a ContextVar in a context manager:
_setting = contextvars.ContextVar('setting')
@contextlib.contextmanager
def setting(value):
tok = _setting.set(value)
try:
yield
finally:
_setting.reset(tok)
and later:
with setting(something): ...
Yury
On Wed, Jun 5, 2019 at 11:19 AM Christoph Groth
It seems that the following idea has not been raised here before. If it has been, I'd be grateful for a pointer to the relevant discussion.
The idea is: how about adding context manager interface (__enter__ and __exit__ methods) to the contextvars.Token type?
This would allow code like this:
with var.set(1): # var.get() == 1
that would be equivalent to the following more verbose snippet (taken from PEP 567):
token = var.set(1) try: # var.get() == 1 finally: var.reset(token)
I attach a very rough proof-of-concept implementation of the idea. The proper way to implement this proposal would be of course to modify the _contextvars C-extension.
Here is my motivation for this proposal:
The contextvars module is promoted as a replacement for thread-local storage for asynchronous programming, but in fact it seems to me that ContextVars implement thread-safe and async-safe dynamic scoping [1] in Python.
One of the uses of dynamic scoping is as an alternative to function parameters for library configuration [2]. While dynamic scoping by default (as in Emacs Lisp) can be dangerous, I believe that explicit dynamic scoping could be useful in the Python world as a means to avoid the "parameter hell" of some libraries, perhaps most infamously demonstrated by matplotlib. With the updated ContextVars, a hypothetical plotting library could be used like this:
with plotting.line_thickness.set(2), plotting.line_style('dashed'): plotting.plot(x, y)
As explained in [2], the advantages of this approach compared to argument passing is that other functions that internally use plotting.plot do not have to expose all of its configuration options by themselves as parameters. Also, with the same mechanism it is possible to set a parameter for a single invocation of plotting.plot, or a default value for a whole script.
[1] https://en.wikipedia.org/wiki/Scope_(computer_science)#Dynamic_scoping [2] https://www.gnu.org/software/emacs/emacs-paper.html#SEC18
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s Code of Conduct: http://python.org/psf/codeofconduct/
-- Yury
Hi Yury, thanks for the quick reply. Yury Selivanov wrote:
Adding context manager protocol support to contextvars.Token was considered when PEP 567 was discussed. There wasn't a strong argument against that; however we decided not to immediately add it because context variables is a relatively low-level API. In you case, you can simply wrap a ContextVar in a context manager:
(...)
I'm aware of this possibility, however it is not suitable for the use case that I have in mind (configuration variables), because it requires too much code for each variable. Of course one could wrap your example inside a factory function, but that's unwieldy as well, since it requires keeping track of one context manager for each context variable: def factory(name): _setting = contextvars.ContextVar(name) @contextlib.contextmanager def setting(value): tok = _setting.set(value) try: yield finally: _setting.reset(tok) return _setting, setting One possibility to imlement this cleanly and efficiently without modifying the standard library module would be to derive classes from ContextVar and from Token, but this is not possible. So it seems to me that if the use of context variables as explicit dynamically bound variables for library configuration is considered interesting, one should think about adding __enter__ and __exit__ to contextvars.Token. By the way, contexts being implemented as immutable dictionaries implies one copy of a dictionary per call to the set method. That in turn means that using hundreds of context variables (a large library could easily have that many configuration variables) might not be a good idea. Is this correct? A final, unrelated comment: I find the use of the word "context" in the standard library for both context managers and context variables (and related to them "contexts") needlessly confusing. I suggest clearly explaining their independence at the top of the documentation of either module.
On Wed, Jun 5, 2019 at 6:22 PM Christoph Groth
I'm aware of this possibility, however it is not suitable for the use case that I have in mind (configuration variables), because it requires too much code for each variable.
Of course one could wrap your example inside a factory function, but that's unwieldy as well, since it requires keeping track of one context manager for each context variable:
I suggest you to open an issue on bugs.python.org to implement support for context manager protocol for contextvars.Token. I'm not opposed to the idea. Keep in mind that Python 3.8 is already in a feature freeze mode, so the earliest we can get this is Python 3.9.
By the way, contexts being implemented as immutable dictionaries implies one copy of a dictionary per call to the set method. That in turn means that using hundreds of context variables (a large library could easily have that many configuration variables) might not be a good idea. Is this correct?
The contextvars module uses a special dictionary implementation (read more on that in PEP 567/550) with its ".set()" operation only slightly slower than updating a standard Python dict. Setting/resetting N context variables is about 1.5x slower than mutating a Python dict N times. In short, it's a fast operation by Python standards.
A final, unrelated comment: I find the use of the word "context" in the standard library for both context managers and context variables (and related to them "contexts") needlessly confusing. I suggest clearly explaining their independence at the top of the documentation of either module.
Pull requests to improve the docs are always welcome! Yury
I had a similar recent need, with a bit more on top of it, and solved it
with this slightly insane library. (Alas, I haven't figured out a good way
to make it act as a true subtype of UnderlyingType yet)
import contextvars
from typing import Any, Generic, TypeVar
UnderlyingType = TypeVar('UnderlyingType')
class DuckTypeContextVar(Generic[UnderlyingType]):
"""A DuckTypeContextVar takes contextvars.ContextVar to the next level:
It duck-type emulates
the underlying variable. This means you can do something like
sys.stdout = DuckTypeContextVar('stdout', sys.stdout)
token = sys.stdout.setLocalValue(myFile)
....
sys.stdout.resetLocalValue(token)
"""
def __init__(self, name: str, value: UnderlyingType) -> None:
self._value = contextvars.ContextVar(name, default=value)
def getLocalValue(self) -> UnderlyingType:
return self._value.get()
def setLocalValue(self, value: UnderlyingType) -> contextvars.Token:
return self._value.set(value)
def resetLocalValue(self, token: contextvars.Token) -> None:
self._value.reset(token)
def __getattr__(self, attr: str) -> Any:
return getattr(self._value, attr)
On Wed, Jun 5, 2019 at 6:14 PM Yury Selivanov
On Wed, Jun 5, 2019 at 6:22 PM Christoph Groth
wrote: [..] I'm aware of this possibility, however it is not suitable for the use case that I have in mind (configuration variables), because it requires too much code for each variable.
Of course one could wrap your example inside a factory function, but that's unwieldy as well, since it requires keeping track of one context manager for each context variable:
I suggest you to open an issue on bugs.python.org to implement support for context manager protocol for contextvars.Token. I'm not opposed to the idea. Keep in mind that Python 3.8 is already in a feature freeze mode, so the earliest we can get this is Python 3.9.
By the way, contexts being implemented as immutable dictionaries implies one copy of a dictionary per call to the set method. That in turn means that using hundreds of context variables (a large library could easily have that many configuration variables) might not be a good idea. Is this correct?
The contextvars module uses a special dictionary implementation (read more on that in PEP 567/550) with its ".set()" operation only slightly slower than updating a standard Python dict. Setting/resetting N context variables is about 1.5x slower than mutating a Python dict N times. In short, it's a fast operation by Python standards.
A final, unrelated comment: I find the use of the word "context" in the standard library for both context managers and context variables (and related to them "contexts") needlessly confusing. I suggest clearly explaining their independence at the top of the documentation of either module.
Pull requests to improve the docs are always welcome!
Yury Python-Ideas mailing list -- python-dev(a)python.org To unsubscribe send an email to python-ideas-leave(a)python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/
Yonatan Zunger wrote:
I had a similar recent need, with a bit more on top of it, and solved it with this slightly insane library. (Alas, I haven't figured out a good way to make it act as a true subtype of UnderlyingType yet)
(...)
To me, your trick seems to address a different problem. You want context variables to be transparent and behave themselves as the data that they hold (in C++ terms: you want reference semantics instead of pointer semantics). This trick might be necessary at some times (like your example of replacing sys.stdout with a context var), but I consider it dangerous for general usage.
On Thu, Jun 6, 2019 at 4:15 AM Christoph Groth
Yonatan Zunger wrote:
I had a similar recent need, with a bit more on top of it, and solved it with this slightly insane library. (Alas, I haven't figured out a good way to make it act as a true subtype of UnderlyingType yet)
(...)
To me, your trick seems to address a different problem. You want context variables to be transparent and behave themselves as the data that they hold (in C++ terms: you want reference semantics instead of pointer semantics).
This trick might be necessary at some times (like your example of replacing sys.stdout with a context var), but I consider it dangerous for general usage.
Oh, absolutely. There's a reason I wasn't proposing that this be added to any standard library. :) That said, it (and other things) have made me think that it would be very nice to have a better way of expressing duck-type inheritance in the typing system that doesn't rely on the superclass declaring itself in particular ways, but that's an entirely separate issue and one I haven't thought through in any detail yet.
Yury Selivanov wrote:
I suggest you to open an issue on bugs.python.org to implement support for context manager protocol for contextvars.Token. I'm not opposed to the idea. Keep in mind that Python 3.8 is already in a feature freeze mode, so the earliest we can get this is Python 3.9.
I'll do that. Le me just raise one more aspect: for the usage of context vars as configuration variables (and likely for other uses as well), it would be useful if there was some way to validate new values when they are set. This would provide immediate feedback to the user and would avoid having to check them at every use. Did you consider anything like that? For example, ContextVar could accept an optional keyword arg 'validate' that must be a function that is then called for each new value, and somehow reports problems (by return value or by raising an exception). (A somewhat related issue would be support for ContextVars in the typing module.)
The contextvars module uses a special dictionary implementation (read more on that in PEP 567/550) with its ".set()" operation only slightly slower than updating a standard Python dict. Setting/resetting N context variables is about 1.5x slower than mutating a Python dict N times. In short, it's a fast operation by Python standards.
Thanks.
A final, unrelated comment: I find the use of the word "context" in the standard library for both context managers and context variables (and related to them "contexts") needlessly confusing. (...)
Pull requests to improve the docs are always welcome!
Sure, I just wanted to double check whether I'm overseeing anything here or whether both kinds of contexts are indeed completely independent concepts.
On Thu, Jun 6, 2019 at 8:04 AM Christoph Groth
Yury Selivanov wrote:
I suggest you to open an issue on bugs.python.org to implement support for context manager protocol for contextvars.Token. I'm not opposed to the idea. Keep in mind that Python 3.8 is already in a feature freeze mode, so the earliest we can get this is Python 3.9.
I'll do that. Le me just raise one more aspect: for the usage of context vars as configuration variables (and likely for other uses as well), it would be useful if there was some way to validate new values when they are set. This would provide immediate feedback to the user and would avoid having to check them at every use.
Did you consider anything like that? For example, ContextVar could accept an optional keyword arg 'validate' that must be a function that is then called for each new value, and somehow reports problems (by return value or by raising an exception).
For that you should definitely use a wrapper. Let's use composition instead of complicating the existing APIs.
(A somewhat related issue would be support for ContextVars in the typing module.)
contextvars should already support this syntax: c: ContextVar[int] = ContextVar('c') Now it's up to mypy et al to support this if they don't already (I haven't checked that.) [..]
A final, unrelated comment: I find the use of the word "context" in the standard library for both context managers and context variables (and related to them "contexts") needlessly confusing. (...)
Pull requests to improve the docs are always welcome!
Sure, I just wanted to double check whether I'm overseeing anything here or whether both kinds of contexts are indeed completely independent concepts.
They aren't completely independent, as it's quite useful to store context manager's state in a context variable (decimal context is a great example of that). OTOH I can totally understand why people can be confused about this. I would love to improve the docs to lessen the confusion where possible, but we're going to keep using the term for both "context managers" and "context variables" as it's too late to change that. -- Yury
Yury Selivanov wrote:
Christoph Groth
wrote: Did you consider anything like that? For example, ContextVar could accept an optional keyword arg 'validate' that must be a function that is then called for each new value, and somehow reports problems (by return value or by raising an exception).
For that you should definitely use a wrapper. Let's use composition instead of complicating the existing APIs.
Agree in principle, but how would that work? If it was possible to derive a Python class from ContextVar, one could add the checking easily and transparently. But unfortunately that's not possible (perhaps it could be fixed?). The other possibility I see would be wrapping ContextVar inside a class (like in the example module that I attached to the first post in this thread), but that's not a good solution. It requires replicating the complete API, and is hence slow and not forward-compatible.
On Thu, Jun 6, 2019 at 5:20 PM Christoph Groth
The other possibility I see would be wrapping ContextVar inside a class (like in the example module that I attached to the first post in this thread), but that's not a good solution. It requires replicating the complete API, and is hence slow and not forward-compatible.
Wrapping isn't much slower than subclassing and calling super(). Indeed, you'll need to write some code that forwards calls to lower-level contextvars APIs, but that's fine. Enhancing contextvars with validators to satisfy your very specific use case would make contextvars slightly slower for all users that don't need that. I'm afraid that wrappers (composition) is the only way here. Yury
Yury Selivanov wrote:
On Thu, Jun 6, 2019 at 5:20 PM Christoph Groth
wrote: [..] The other possibility I see would be wrapping ContextVar inside a class (like in the example module that I attached to the first post in this thread), but that's not a good solution. It requires replicating the complete API, and is hence slow and not forward-compatible.
Wrapping isn't much slower than subclassing and calling super(). (...)
What is actually the reason that Py_TPFLAGS_BASETYPE is not set for ContextVar (and many other built-in types)? The performance penality cannot be significant since both tuple and list are subclassable. Does it complicate the implementation? The requirements [1] do not look bad. Am I missing anything obvious here? (The extension types that I wrote are also not subclassable, but I just followed the default here and so far no one complained.) [1] https://www.python.org/dev/peps/pep-0253/#preparing-a-type-for-subtyping
participants (3)
-
Christoph Groth
-
Yonatan Zunger
-
Yury Selivanov