Hi, This is a third version of PEP 567. Changes from v2: 1. PyThreadState now references Context objects directly (instead of referencing _ContextData). This fixes out of sync Context.get() and ContextVar.get(). 2. Added a new Context.copy() method. 3. Renamed Token.old_val property to Token.old_value 4. ContextVar.reset(token) now raises a ValueError if the token was created in a different Context. 5. All areas of the PEP were updated to be more precise. Context is *no longer* defined as a read-only or an immutable mapping; ContextVar.get() behaviour is fully defined; the immutability is only mentioned in the Implementation section to avoid confusion; etc. 6. Added a new Examples section. The reference implementation has been updated to include all these changes. The only open question I personally have is whether ContextVar.reset() should be idempotent or not. Maybe we should be strict and raise an error if a user tries to reset a variable more than once with the same token object? Other than that, I'm pretty happy with this version. Big thanks to everybody helping with the PEP! PEP: 567 Title: Context Variables Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov <yury@magic.io> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Dec-2017 Python-Version: 3.7 Post-History: 12-Dec-2017, 28-Dec-2017, 16-Jan-2018 Abstract ======== This PEP proposes a new ``contextvars`` module and a set of new CPython C APIs to support context variables. This concept is similar to thread-local storage (TLS), but, unlike TLS, it also allows correctly keeping track of values per asynchronous task, e.g. ``asyncio.Task``. This proposal is a simplified version of :pep:`550`. The key difference is that this PEP is concerned only with solving the case for asynchronous tasks, not for generators. There are no proposed modifications to any built-in types or to the interpreter. This proposal is not strictly related to Python Context Managers. Although it does provide a mechanism that can be used by Context Managers to store their state. Rationale ========= Thread-local variables are insufficient for asynchronous tasks that execute concurrently in the same OS thread. Any context manager that saves and restores a context value using ``threading.local()`` will have its context values bleed to other code unexpectedly when used in async/await code. A few examples where having a working context local storage for asynchronous code is desirable: * Context managers like ``decimal`` contexts and ``numpy.errstate``. * Request-related data, such as security tokens and request data in web applications, language context for ``gettext``, etc. * Profiling, tracing, and logging in large code bases. Introduction ============ The PEP proposes a new mechanism for managing context variables. The key classes involved in this mechanism are ``contextvars.Context`` and ``contextvars.ContextVar``. The PEP also proposes some policies for using the mechanism around asynchronous tasks. The proposed mechanism for accessing context variables uses the ``ContextVar`` class. A module (such as ``decimal``) that wishes to use the new mechanism should: * declare a module-global variable holding a ``ContextVar`` to serve as a key; * access the current value via the ``get()`` method on the key variable; * modify the current value via the ``set()`` method on the key variable. The notion of "current value" deserves special consideration: different asynchronous tasks that exist and execute concurrently may have different values for the same key. This idea is well-known from thread-local storage but in this case the locality of the value is not necessarily bound to a thread. Instead, there is the notion of the "current ``Context``" which is stored in thread-local storage. Manipulation of the current context is the responsibility of the task framework, e.g. asyncio. A ``Context`` is a mapping of ``ContextVar`` objects to their values. The ``Context`` itself exposes the ``abc.Mapping`` interface (not ``abc.MutableMapping``!), so it cannot be modified directly. To set a new value for a context variable in a ``Context`` object, the user needs to: * make the ``Context`` object "current" using the ``Context.run()`` method; * use ``ContextVar.set()`` to set a new value for the context variable. The ``ContextVar.get()`` method looks for the variable in the current ``Context`` object using ``self`` as a key. It is not possible to get a direct reference to the current ``Context`` object, but it is possible to obtain a shallow copy of it using the ``contextvars.copy_context()`` function. This ensures that the caller of ``Context.run()`` is the sole owner of its ``Context`` object. Specification ============= A new standard library module ``contextvars`` is added with the following APIs: 1. ``copy_context() -> Context`` function is used to get a copy of the current ``Context`` object for the current OS thread. 2. ``ContextVar`` class to declare and access context variables. 3. ``Context`` class encapsulates context state. Every OS thread stores a reference to its current ``Context`` instance. It is not possible to control that reference directly. Instead, the ``Context.run(callable, *args, **kwargs)`` method is used to run Python code in another context. contextvars.ContextVar ---------------------- The ``ContextVar`` class has the following constructor signature: ``ContextVar(name, *, default=_NO_DEFAULT)``. The ``name`` parameter is used for introspection and debug purposes, and is exposed as a read-only ``ContextVar.name`` attribute. The ``default`` parameter is optional. Example:: # Declare a context variable 'var' with the default value 42. var = ContextVar('var', default=42) (The ``_NO_DEFAULT`` is an internal sentinel object used to detect if the default value was provided.) ``ContextVar.get(default=_NO_DEFAULT)`` returns a value for the context variable for the current ``Context``:: # Get the value of `var`. var.get() If there is no value for the variable in the current context, ``ContextVar.get()`` will: * return the value of the *default* argument of the ``get()`` method, if provided; or * return the default value for the context variable, if provided; or * raise a ``LookupError``. ``ContextVar.set(value) -> Token`` is used to set a new value for the context variable in the current ``Context``:: # Set the variable 'var' to 1 in the current context. var.set(1) ``ContextVar.reset(token)`` is used to reset the variable in the current context to the value it had before the ``set()`` operation that created the ``token`` (or to remove the variable if it was not set):: assert var.get(None) is None token = var.set(1) try: ... finally: var.reset(token) assert var.get(None) is None ``ContextVar.reset()`` method is idempotent and can be called multiple times on the same Token object: second and later calls will be no-ops. The method raises a ``ValueError`` if: * called with a token object created by another variable; or * the current ``Context`` object does not match the one where the token object was created. contextvars.Token ----------------- ``contextvars.Token`` is an opaque object that should be used to restore the ``ContextVar`` to its previous value, or to remove it from the context if the variable was not set before. It can be created only by calling ``ContextVar.set()``. For debug and introspection purposes it has: * a read-only attribute ``Token.var`` pointing to the variable that created the token; * a read-only attribute ``Token.old_value`` set to the value the variable had before the ``set()`` call, or to ``Token.MISSING`` if the variable wasn't set before. contextvars.Context ------------------- ``Context`` object is a mapping of context variables to values. ``Context()`` creates an empty context. To get a copy of the current ``Context`` for the current OS thread, use the ``contextvars.copy_context()`` method:: ctx = contextvars.copy_context() To run Python code in some ``Context``, use ``Context.run()`` method:: ctx.run(function) Any changes to any context variables that ``function`` causes will be contained in the ``ctx`` context:: var = ContextVar('var') var.set('spam') def function(): assert var.get() == 'spam' assert ctx[var] == 'spam' var.set('ham') assert var.get() == 'ham' assert ctx[var] == 'ham' ctx = copy_context() # Any changes that 'function' makes to 'var' will stay # isolated in the 'ctx'. ctx.run(function) assert var.get() == 'spam' assert ctx[var] == 'ham' ``Context.run()`` raises a ``RuntimeError`` when called on the same context object from more than one OS thread, or when called recursively. ``Context.copy()`` returns a shallow copy of the context object. ``Context`` objects implement the ``collections.abc.Mapping`` ABC. This can be used to introspect contexts:: ctx = contextvars.copy_context() # Print all context variables and their values in 'ctx': print(ctx.items()) # Print the value of 'some_variable' in context 'ctx': print(ctx[some_variable]) Note that all Mapping methods, including ``Context.__getitem__`` and ``Context.get``, ignore default values for context variables (i.e. ``ContextVar.default``). This means that for a variable *var* that was created with a default value and was not set in the *context*: * ``context[var]`` raises a ``KeyError``, * ``var in context`` returns ``False``, * the variable isn't included in ``context.items()``, etc. asyncio ------- ``asyncio`` uses ``Loop.call_soon()``, ``Loop.call_later()``, and ``Loop.call_at()`` to schedule the asynchronous execution of a function. ``asyncio.Task`` uses ``call_soon()`` to run the wrapped coroutine. We modify ``Loop.call_{at,later,soon}`` and ``Future.add_done_callback()`` to accept the new optional *context* keyword-only argument, which defaults to the current context:: def call_soon(self, callback, *args, context=None): if context is None: context = contextvars.copy_context() # ... some time later context.run(callback, *args) Tasks in asyncio need to maintain their own context that they inherit from the point they were created at. ``asyncio.Task`` is modified as follows:: class Task: def __init__(self, coro): ... # Get the current context snapshot. self._context = contextvars.copy_context() self._loop.call_soon(self._step, context=self._context) def _step(self, exc=None): ... # Every advance of the wrapped coroutine is done in # the task's context. self._loop.call_soon(self._step, context=self._context) ... Implementation ============== This section explains high-level implementation details in pseudo-code. Some optimizations are omitted to keep this section short and clear. The ``Context`` mapping is implemented using an immutable dictionary. This allows for a O(1) implementation of the ``copy_context()`` function. The reference implementation implements the immutable dictionary using Hash Array Mapped Tries (HAMT); see :pep:`550` for analysis of HAMT performance [1]_. For the purposes of this section, we implement an immutable dictionary using a copy-on-write approach and built-in dict type:: class _ContextData: def __init__(self): self._mapping = dict() def get(self, key): return self._mapping[key] def set(self, key, value): copy = _ContextData() copy._mapping = self._mapping.copy() copy._mapping[key] = value return copy def delete(self, key): copy = _ContextData() copy._mapping = self._mapping.copy() del copy._mapping[key] return copy Every OS thread has a reference to the current ``Context`` object:: class PyThreadState: context: Context ``contextvars.Context`` is a wrapper around ``_ContextData``:: class Context(collections.abc.Mapping): _data: _ContextData _prev_context: Optional[Context] def __init__(self): self._data = _ContextData() self._prev_context = None def run(self, callable, *args, **kwargs): if self._prev_context is not None: raise RuntimeError( f'cannot enter context: {self} is already entered') ts: PyThreadState = PyThreadState_Get() if ts.context is None: ts.context = Context() self._prev_context = ts.context try: ts.context = self return callable(*args, **kwargs) finally: ts.context = self._prev_context self._prev_context = None def copy(self): new = Context() new._data = self._data return new # Mapping API methods are implemented by delegating # `get()` and other Mapping methods to `self._data`. ``contextvars.copy_context()`` is implemented as follows:: def copy_context(): ts: PyThreadState = PyThreadState_Get() if ts.context is None: ts.context = Context() return ts.context.copy() ``contextvars.ContextVar`` interacts with ``PyThreadState.context`` directly:: class ContextVar: def __init__(self, name, *, default=_NO_DEFAULT): self._name = name self._default = default @property def name(self): return self._name def get(self, default=_NO_DEFAULT): ts: PyThreadState = PyThreadState_Get() if ts.context is not None: try: return ts.context[self] except KeyError: pass if default is not _NO_DEFAULT: return default if self._default is not _NO_DEFAULT: return self._default raise LookupError def set(self, value): ts: PyThreadState = PyThreadState_Get() if ts.context is None: ts.context = Context() data: _ContextData = ts.context._data try: old_value = data.get(self) except KeyError: old_value = Token.MISSING updated_data = data.set(self, value) ts.context._data = updated_data return Token(ts.context, self, old_value) def reset(self, token): if token._var is not self: raise ValueError( "Token was created by a different ContextVar") ts: PyThreadState = PyThreadState_Get() if token._ctx is not ts.context: raise ValueError( "Token was created in a different Context") if token._used: return if token._old_value is Token.MISSING: ts.context._data = data.delete(token._var) else: ts.context._data = data.set(token._var, token._old_value) token._used = True Note that the in the reference implementation, ``ContextVar.get()`` has an internal cache for the most recent value, which allows to bypass a hash lookup. This is similar to the optimization the ``decimal`` module implements to retrieve its context from ``PyThreadState_GetDict()``. See :pep:`550` which explains the implementation of the cache in great detail. The ``Token`` class is implemented as follows:: class Token: MISSING = object() def __init__(self, ctx, var, old_value): self._ctx = ctx self._var = var self._old_value = old_value self._used = False @property def var(self): return self._var @property def old_value(self): return self._old_value Summary of the New APIs ======================= Python API ---------- 1. A new ``contextvars`` module with ``ContextVar``, ``Context``, and ``Token`` classes, and a ``copy_context()`` function. 2. ``asyncio.Loop.call_at()``, ``asyncio.Loop.call_later()``, ``asyncio.Loop.call_soon()``, and ``asyncio.Future.add_done_callback()`` run callback functions in the context they were called in. A new *context* keyword-only parameter can be used to specify a custom context. 3. ``asyncio.Task`` is modified internally to maintain its own context. C API ----- 1. ``PyContextVar * PyContextVar_New(char *name, PyObject *default)``: create a ``ContextVar`` object. The *default* argument can be ``NULL``, which means that the variable has no default value. 2. ``int PyContextVar_Get(PyContextVar *, PyObject *default_value, PyObject **value)``: return ``-1`` if an error occurs during the lookup, ``0`` otherwise. If a value for the context variable is found, it will be set to the ``value`` pointer. Otherwise, ``value`` will be set to ``default_value`` when it is not ``NULL``. If ``default_value`` is ``NULL``, ``value`` will be set to the default value of the variable, which can be ``NULL`` too. ``value`` is always a new reference. 3. ``PyContextToken * PyContextVar_Set(PyContextVar *, PyObject *)``: set the value of the variable in the current context. 4. ``PyContextVar_Reset(PyContextVar *, PyContextToken *)``: reset the value of the context variable. 5. ``PyContext * PyContext_New()``: create a new empty context. 6. ``PyContext * PyContext_Copy()``: get a copy of the current context. 7. ``int PyContext_Enter(PyContext *)`` and ``int PyContext_Exit(PyContext *)`` allow to set and restore the context for the current OS thread. It is required to always restore the previous context:: PyContext *old_ctx = PyContext_Copy(); if (old_ctx == NULL) goto error; if (PyContext_Enter(new_ctx)) goto error; // run some code if (PyContext_Exit(old_ctx)) goto error; Design Considerations ===================== Why contextvars.Token and not ContextVar.unset()? ------------------------------------------------- The Token API allows to get around having a ``ContextVar.unset()`` method, which is incompatible with chained contexts design of :pep:`550`. Future compatibility with :pep:`550` is desired (at least for Python 3.7) in case there is demand to support context variables in generators and asynchronous generators. The Token API also offers better usability: the user does not have to special-case absence of a value. Compare:: token = cv.get() try: cv.set(blah) # code finally: cv.reset(token) with:: _deleted = object() old = cv.get(default=_deleted) try: cv.set(blah) # code finally: if old is _deleted: cv.unset() else: cv.set(old) Rejected Ideas ============== Replication of threading.local() interface ------------------------------------------ Please refer to :pep:`550` where this topic is covered in detail: [2]_. Backwards Compatibility ======================= This proposal preserves 100% backwards compatibility. Libraries that use ``threading.local()`` to store context-related values, currently work correctly only for synchronous code. Switching them to use the proposed API will keep their behavior for synchronous code unmodified, but will automatically enable support for asynchronous code. Examples ======== Converting code that uses threading.local() ------------------------------------------- A typical code that uses ``threading.local()`` usually looks like the following snippet:: class mylocal(threading.local): # Subclass threading.local to specify a default value. value = 'spam' mylocal = mylocal() # To set a new value: mylocal.value = 'new value' # To read the current value: mylocal.value Such code can be converted to use the ``contextvars`` module:: mylocal = contextvars.ContextVar('mylocal', 'spam') # To set a new value: mylocal.set('new value') # To read the current value: mylocal.get() Offloading execution to other threads ------------------------------------- It is possible to run code in a separate OS thread using a copy of the current thread context:: executor = ThreadPoolExecutor() current_context = contextvars.copy_context() executor.submit( lambda: current_context.run(some_function)) Reference Implementation ======================== The reference implementation can be found here: [3]_. References ========== .. [1] https://www.python.org/dev/peps/pep-0550/#appendix-hamt-performance-analysis .. [2] https://www.python.org/dev/peps/pep-0550/#replication-of-threading-local-int... .. [3] https://github.com/python/cpython/pull/5027 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
Hi Yury, Thanks for the updated PEP v3, it's now much better than PEP v2! I have no more complain against your PEP. I vote +1 for PEP 567 contextvars!
The only open question I personally have is whether ContextVar.reset() should be idempotent or not. Maybe we should be strict and raise an error if a user tries to reset a variable more than once with the same token object?
I don't think that it's worth it to prevent misuage of reset(). IMHO it's fine if calling reset() twice reverts the variable state twice. Victor
On Tue, Jan 16, 2018 at 3:26 PM, Victor Stinner <victor.stinner@gmail.com> wrote:
Thanks for the updated PEP v3, it's now much better than PEP v2!
I have no more complain against your PEP. I vote +1 for PEP 567 contextvars!
Yeah!
The only open question I personally have is whether ContextVar.reset()
should be idempotent or not. Maybe we should be strict and raise an error if a user tries to reset a variable more than once with the same token object?
I don't think that it's worth it to prevent misuage of reset(). IMHO it's fine if calling reset() twice reverts the variable state twice.
Maybe the effect of calling it twice should be specified as undefined -- the implementation can try to raise in simple cases. Unless Yury has a use case for the idempotency? (But with __enter__/__exit__ as the main use case for reset() I wouldn't know what the use case for idempotency would be.) -- --Guido van Rossum (python.org/~guido)
On Tue, 16 Jan 2018 17:44:14 -0500 Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Offloading execution to other threads -------------------------------------
It is possible to run code in a separate OS thread using a copy of the current thread context::
executor = ThreadPoolExecutor() current_context = contextvars.copy_context()
executor.submit( lambda: current_context.run(some_function))
Does it also support offloading to a separate process (using ProcessPoolExecutor in the example above)? This would require the Context to support pickling. Regards Antoine.
On Tue, Jan 16, 2018 at 4:37 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Tue, 16 Jan 2018 17:44:14 -0500 Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Offloading execution to other threads -------------------------------------
It is possible to run code in a separate OS thread using a copy of the current thread context::
executor = ThreadPoolExecutor() current_context = contextvars.copy_context()
executor.submit( lambda: current_context.run(some_function))
Does it also support offloading to a separate process (using ProcessPoolExecutor in the example above)? This would require the Context to support pickling.
I don't think that's a requirement. The transparency between the two different types of executor is mostly misleading anyway -- it's like the old RPC transparency problem, which was never solved IIRC. There are just too many things you need to be aware of before you can successfully offload something to a different process. -- --Guido van Rossum (python.org/~guido)
On Tue, Jan 16, 2018 at 6:53 PM, Guido van Rossum <guido@python.org> wrote:
On Tue, Jan 16, 2018 at 3:26 PM, Victor Stinner <victor.stinner@gmail.com> [..]
I don't think that it's worth it to prevent misuage of reset(). IMHO it's fine if calling reset() twice reverts the variable state twice.
Maybe the effect of calling it twice should be specified as undefined -- the implementation can try to raise in simple cases.
Unless Yury has a use case for the idempotency? (But with __enter__/__exit__ as the main use case for reset() I wouldn't know what the use case for idempotency would be.)
I don't have any use case for idempotent reset, so I'd change it to raise an error on second call. We can always relax this in 3.8 if people request it to be idempotent. Yury
On Tue, Jan 16, 2018 at 7:45 PM, Guido van Rossum <guido@python.org> wrote:
On Tue, Jan 16, 2018 at 4:37 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Tue, 16 Jan 2018 17:44:14 -0500 Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Offloading execution to other threads -------------------------------------
It is possible to run code in a separate OS thread using a copy of the current thread context::
executor = ThreadPoolExecutor() current_context = contextvars.copy_context()
executor.submit( lambda: current_context.run(some_function))
Does it also support offloading to a separate process (using ProcessPoolExecutor in the example above)? This would require the Context to support pickling.
I don't think that's a requirement. The transparency between the two different types of executor is mostly misleading anyway -- it's like the old RPC transparency problem, which was never solved IIRC. There are just too many things you need to be aware of before you can successfully offload something to a different process.
I agree. I think it would be a very fragile thing In practice: if you have even one variable in the context that isn't pickleable, your code that uses a ProcessPool would stop working. I would defer Context pickleability to 3.8+. Yury
On Tue, Jan 16, 2018 at 5:06 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
On Tue, Jan 16, 2018 at 7:45 PM, Guido van Rossum <guido@python.org> wrote:
On Tue, Jan 16, 2018 at 4:37 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Tue, 16 Jan 2018 17:44:14 -0500 Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Offloading execution to other threads -------------------------------------
It is possible to run code in a separate OS thread using a copy of the current thread context::
executor = ThreadPoolExecutor() current_context = contextvars.copy_context()
executor.submit( lambda: current_context.run(some_function))
Does it also support offloading to a separate process (using ProcessPoolExecutor in the example above)? This would require the Context to support pickling.
I don't think that's a requirement. The transparency between the two different types of executor is mostly misleading anyway -- it's like the old RPC transparency problem, which was never solved IIRC. There are just too many things you need to be aware of before you can successfully offload something to a different process.
I agree.
I think it would be a very fragile thing In practice: if you have even one variable in the context that isn't pickleable, your code that uses a ProcessPool would stop working. I would defer Context pickleability to 3.8+.
There's also a more fundamental problem: you need some way to match up the ContextVar objects across the two processes, and right now they don't have an attached __module__ or __qualname__. I guess we could do like namedtuple and (a) capture the module where the ContextVar was instantiated, on the assumption that that's where it will be stored, (b) require that users pass in the name of variable where it will be stored as the 'name' argument to ContextVar.__init__. I tend to agree that this is something to worry about for 3.8 though. (If we need to retrofit pickle support, we could add a pickleable=False argument to ContextVar, and require people to pass pickleable=True to signal that they've done the appropriate setup to make the ContextVar identifiable across processes, and that its contents are safe to pickle.) -n -- Nathaniel J. Smith -- https://vorpus.org
On Tue, Jan 16, 2018 at 2:44 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
4. ContextVar.reset(token) now raises a ValueError if the token was created in a different Context.
A minor bit of polish: given that Token objects have to track the associated ContextVar anyway, I think it'd be cleaner if instead of doing: token = cvar.set(...) cvar.reset(token) we made the API be: token = cvar.set(...) token.reset() In the first version, we use 'cvar' twice, and it's a mandatory invariant that the same ContextVar object gets used in both places; you had to add extra code to check this and raise an error if that's violated. It's level 5 on Rusty's scale (http://sweng.the-davies.net/Home/rustys-api-design-manifesto) In the second version, the ContextVar is only mentioned once, so the invariant is automatically enforced by the API -- you can't even express the broken version. That's level 10 on Rusty's scale, and gives a simpler implementation too. -n -- Nathaniel J. Smith -- https://vorpus.org
On Tue, Jan 16, 2018 at 8:27 PM, Nathaniel Smith <njs@pobox.com> wrote: [..]
token = cvar.set(...) token.reset()
I see the point, but I think that having the 'reset' method defined on the ContextVar class is easier to grasp. It also feels natural that a pair of set/reset methods is defined on the same class. This is highly subjective though, so let's see which option Guido likes more. Yury
On Tue, Jan 16, 2018 at 5:33 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
On Tue, Jan 16, 2018 at 8:27 PM, Nathaniel Smith <njs@pobox.com> wrote: [..]
token = cvar.set(...) token.reset()
I see the point, but I think that having the 'reset' method defined on the ContextVar class is easier to grasp. It also feels natural that a pair of set/reset methods is defined on the same class. This is highly subjective though, so let's see which option Guido likes more.
I think this came up in one of the previous reviews of the PEP. I like Yury's (redundant) version -- it makes it clear to the human reader of the code which variable is being reset. And it's not like it's going to be used that much -- it'll be likely hidden inside a context manager. -- --Guido van Rossum (python.org/~guido)
On 17 January 2018 at 11:27, Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Jan 16, 2018 at 2:44 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
4. ContextVar.reset(token) now raises a ValueError if the token was created in a different Context.
A minor bit of polish: given that Token objects have to track the associated ContextVar anyway, I think it'd be cleaner if instead of doing:
token = cvar.set(...) cvar.reset(token)
we made the API be:
token = cvar.set(...) token.reset()
As a counterpoint to this, consider the case where you're working with *two* cvars: token1 = cvar1.set(...) token2 = cvar2.set(...) ... cvar1.reset(token1) ... cvar2.reset(token2) At the point where the resets happen, you know exactly which cvar is being reset, even if you don't know where the token was created. With reset-on-the-token, you're entirely reliant on variable naming to know which ContextVar is going to be affected: token1 = cvar1.set(...) token2 = cvar2.set(...) ... token1.reset() # Resets cvar1 ... token2.reset() # Resets cvar2 If someone really does want an auto-reset API, it's also fairly easy to build atop the more explicit one: def set_cvar(cvar, value): token = cvar.set(value) return functools.partial(cvar.reset, token) reset_cvar1 = set_cvar(cvar1, ...) ... reset_cvar1() Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, 16 Jan 2018 17:18:06 -0800 Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Jan 16, 2018 at 5:06 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
I think it would be a very fragile thing In practice: if you have even one variable in the context that isn't pickleable, your code that uses a ProcessPool would stop working. I would defer Context pickleability to 3.8+.
There's also a more fundamental problem: you need some way to match up the ContextVar objects across the two processes, and right now they don't have an attached __module__ or __qualname__.
They have a name, though. So perhaps the name could serve as a unique identifier? Instead of being serialized as a bunch of ContextVars, the Context would then be serialized as a {name: value} dict. Regards Antoine.
On Wed, Jan 17, 2018 at 6:03 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Tue, 16 Jan 2018 17:18:06 -0800 Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Jan 16, 2018 at 5:06 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
I think it would be a very fragile thing In practice: if you have even one variable in the context that isn't pickleable, your code that uses a ProcessPool would stop working. I would defer Context pickleability to 3.8+.
There's also a more fundamental problem: you need some way to match up the ContextVar objects across the two processes, and right now they don't have an attached __module__ or __qualname__.
They have a name, though. So perhaps the name could serve as a unique identifier? Instead of being serialized as a bunch of ContextVars, the Context would then be serialized as a {name: value} dict.
One of the points of the ContextVar design is to avoid having unique identifiers requirement. Names can clash which leads to data being lost. If you prohibit them from clashing, then if libraries A and B happen to use the same context variable name—you can't use them both in your projects. And without enforcing name uniqueness, your approach to serialize context as a dict with string keys won't work. I like Nathaniel's idea to explicitly enable ContextVars pickling support on a per-var basis. Unfortunately we don't have time to seriously consider and debate (and implement!) this idea in time before the 3.7 freeze. In the meanwhile, given that Context objects are fully introspectable, users can implement their own ad-hoc solutions for serializers or cross-process execution. Yury
Perhaps you can update the PEP with a summary of the rejected ideas from this thread? On Jan 17, 2018 7:23 AM, "Yury Selivanov" <yselivanov.ml@gmail.com> wrote:
On Wed, Jan 17, 2018 at 6:03 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Tue, 16 Jan 2018 17:18:06 -0800 Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Jan 16, 2018 at 5:06 PM, Yury Selivanov < yselivanov.ml@gmail.com> wrote:
I think it would be a very fragile thing In practice: if you have even one variable in the context that isn't pickleable, your code that uses a ProcessPool would stop working. I would defer Context pickleability to 3.8+.
There's also a more fundamental problem: you need some way to match up the ContextVar objects across the two processes, and right now they don't have an attached __module__ or __qualname__.
They have a name, though. So perhaps the name could serve as a unique identifier? Instead of being serialized as a bunch of ContextVars, the Context would then be serialized as a {name: value} dict.
One of the points of the ContextVar design is to avoid having unique identifiers requirement. Names can clash which leads to data being lost. If you prohibit them from clashing, then if libraries A and B happen to use the same context variable name—you can't use them both in your projects. And without enforcing name uniqueness, your approach to serialize context as a dict with string keys won't work.
I like Nathaniel's idea to explicitly enable ContextVars pickling support on a per-var basis. Unfortunately we don't have time to seriously consider and debate (and implement!) this idea in time before the 3.7 freeze.
In the meanwhile, given that Context objects are fully introspectable, users can implement their own ad-hoc solutions for serializers or cross-process execution.
Yury _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ guido%40python.org
FYI In the PEP 540, I didn't try to elaborate on each design change, but I wrote a very short version history at the end: https://www.python.org/dev/peps/pep-0540/#version-history Maybe something like that would help for the PEP 567? Victor Le 17 janv. 2018 8:26 PM, "Guido van Rossum" <gvanrossum@gmail.com> a écrit :
Perhaps you can update the PEP with a summary of the rejected ideas from this thread?
On Jan 17, 2018 7:23 AM, "Yury Selivanov" <yselivanov.ml@gmail.com> wrote:
On Tue, 16 Jan 2018 17:18:06 -0800 Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Jan 16, 2018 at 5:06 PM, Yury Selivanov < yselivanov.ml@gmail.com> wrote:
I think it would be a very fragile thing In practice: if you have
even
one variable in the context that isn't pickleable, your code that uses a ProcessPool would stop working. I would defer Context
On Wed, Jan 17, 2018 at 6:03 AM, Antoine Pitrou <solipsis@pitrou.net> wrote: pickleability
to 3.8+.
There's also a more fundamental problem: you need some way to match up the ContextVar objects across the two processes, and right now they don't have an attached __module__ or __qualname__.
They have a name, though. So perhaps the name could serve as a unique identifier? Instead of being serialized as a bunch of ContextVars, the Context would then be serialized as a {name: value} dict.
One of the points of the ContextVar design is to avoid having unique identifiers requirement. Names can clash which leads to data being lost. If you prohibit them from clashing, then if libraries A and B happen to use the same context variable name—you can't use them both in your projects. And without enforcing name uniqueness, your approach to serialize context as a dict with string keys won't work.
I like Nathaniel's idea to explicitly enable ContextVars pickling support on a per-var basis. Unfortunately we don't have time to seriously consider and debate (and implement!) this idea in time before the 3.7 freeze.
In the meanwhile, given that Context objects are fully introspectable, users can implement their own ad-hoc solutions for serializers or cross-process execution.
Yury _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido% 40python.org
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ victor.stinner%40gmail.com
On Wed, Jan 17, 2018 at 2:24 PM, Guido van Rossum <gvanrossum@gmail.com> wrote:
Perhaps you can update the PEP with a summary of the rejected ideas from this thread?
The Rejected Ideas section of the PEP is now updated with the below: Token.reset() instead of ContextVar.reset() ------------------------------------------- Nathaniel Smith suggested to implement the ``ContextVar.reset()`` method directly on the ``Token`` class, so instead of:: token = var.set(value) # ... var.reset(token) we would write:: token = var.set(value) # ... token.reset() Having ``Token.reset()`` would make it impossible for a user to attempt to reset a variable with a token object created by another variable. This proposal was rejected for the reason of ``ContextVar.reset()`` being clearer to the human reader of the code which variable is being reset. Make Context objects picklable ------------------------------ Proposed by Antoine Pitrou, this could enable transparent cross-process use of ``Context`` objects, so the `Offloading execution to other threads`_ example would work with a ``ProcessPoolExecutor`` too. Enabling this is problematic because of the following reasons: 1. ``ContextVar`` objects do not have ``__module__`` and ``__qualname__`` attributes, making straightforward pickling of ``Context`` objects impossible. This is solvable by modifying the API to either auto detect the module where a context variable is defined, or by adding a new keyword-only "module" parameter to ``ContextVar`` constructor. 2. Not all context variables refer to picklable objects. Making a ``ContextVar`` picklable must be an opt-in. Given the time frame of the Python 3.7 release schedule it was decided to defer this proposal to Python 3.8. Yury
On Wed, Jan 17, 2018 at 8:53 PM, Yury Selivanov <yselivanov.ml@gmail.com> wrote:
On Wed, Jan 17, 2018 at 2:24 PM, Guido van Rossum <gvanrossum@gmail.com> wrote:
Perhaps you can update the PEP with a summary of the rejected ideas from this thread?
The Rejected Ideas section of the PEP is now updated with the below:
I've added two more subsections to Rejected Ideas: Make Context a MutableMapping ----------------------------- Making the ``Context`` class implement the ``abc.MutableMapping`` interface would mean that it is possible to set and unset variables using ``Context[var] = value`` and ``del Context[var]`` operations. This proposal was deferred to Python 3.8+ because of the following: 1. If in Python 3.8 it is decided that generators should support context variables (see :pep:`550` and :pep:`568`), then ``Context`` would be transformed into a chain-map of context variables mappings (as every generator would have its own mapping). That would make mutation operations like ``Context.__delitem__`` confusing, as they would operate only on the topmost mapping of the chain. 2. Having a single way of mutating the context (``ContextVar.set()`` and ``ContextVar.reset()`` methods) makes the API more straightforward. For example, it would be non-obvious why the below code fragment does not work as expected:: var = ContextVar('var') ctx = copy_context() ctx[var] = 'value' print(ctx[var]) # Prints 'value' print(var.get()) # Raises a LookupError While the following code would work:: ctx = copy_context() def func(): ctx[var] = 'value' # Contrary to the previous example, this would work # because 'func()' is running within 'ctx'. print(ctx[var]) print(var.get()) ctx.run(func) Have initial values for ContextVars ----------------------------------- Nathaniel Smith proposed to have a required ``initial_value`` keyword-only argument for the ``ContextVar`` constructor. The main argument against this proposal is that for some types there is simply no sensible "initial value" except ``None``. E.g. consider a web framework that stores the current HTTP request object in a context variable. With the current semantics it is possible to create a context variable without a default value:: # Framework: current_request: ContextVar[Request] = \ ContextVar('current_request') # Later, while handling an HTTP request: request: Request = current_request.get() # Work with the 'request' object: return request.method Note that in the above example there is no need to check if ``request`` is ``None``. It is simply expected that the framework always sets the ``current_request`` variable, or it is a bug (in which case ``current_request.get()`` would raise a ``LookupError``). If, however, we had a required initial value, we would have to guard against ``None`` values explicitly:: # Framework: current_request: ContextVar[Optional[Request]] = \ ContextVar('current_request', initial_value=None) # Later, while handling an HTTP request: request: Optional[Request] = current_request.get() # Check if the current request object was set: if request is None: raise RuntimeError # Work with the 'request' object: return request.method Moreover, we can loosely compare context variables to regular Python variables and to ``threading.local()`` objects. Both of them raise errors on failed lookups (``NameError`` and ``AttributeError`` respectively). Yury
On Wed, 17 Jan 2018 20:53:42 -0500 Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Proposed by Antoine Pitrou, this could enable transparent cross-process use of ``Context`` objects, so the `Offloading execution to other threads`_ example would work with a ``ProcessPoolExecutor`` too.
Enabling this is problematic because of the following reasons:
1. ``ContextVar`` objects do not have ``__module__`` and ``__qualname__`` attributes, making straightforward pickling of ``Context`` objects impossible. This is solvable by modifying the API to either auto detect the module where a context variable is defined, or by adding a new keyword-only "module" parameter to ``ContextVar`` constructor.
2. Not all context variables refer to picklable objects. Making a ``ContextVar`` picklable must be an opt-in.
This is a red herring. If a value isn't picklable, pickle will simply raise as it does in other contexts. You should't need to opt in for anything here. Regards Antoine.
On Thu, Jan 18, 2018 at 12:03 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Wed, 17 Jan 2018 20:53:42 -0500 Yury Selivanov <yselivanov.ml@gmail.com> wrote:
Proposed by Antoine Pitrou, this could enable transparent cross-process use of ``Context`` objects, so the `Offloading execution to other threads`_ example would work with a ``ProcessPoolExecutor`` too.
Enabling this is problematic because of the following reasons:
1. ``ContextVar`` objects do not have ``__module__`` and ``__qualname__`` attributes, making straightforward pickling of ``Context`` objects impossible. This is solvable by modifying the API to either auto detect the module where a context variable is defined, or by adding a new keyword-only "module" parameter to ``ContextVar`` constructor.
2. Not all context variables refer to picklable objects. Making a ``ContextVar`` picklable must be an opt-in.
This is a red herring. If a value isn't picklable, pickle will simply raise as it does in other contexts. You should't need to opt in for anything here.
The complication is that Contexts collect ContextVars from all over the process. So if people are going to pickle Contexts, we need some mechanism to make sure that we don't end up in a situation where it seems to work and users depend on it, and then they import a new library and suddenly pickling raises an error (because the new library internally uses a ContextVar that happens not to be pickleable). -n -- Nathaniel J. Smith -- https://vorpus.org
On Thu, Jan 18, 2018 at 3:53 AM, Yury Selivanov <yselivanov.ml@gmail.com> wrote: [....]
Given the time frame of the Python 3.7 release schedule it was decided to defer this proposal to Python 3.8.
It occurs to me that I had misread this to refer to the whole PEP. Although I thought it's kind of sad that after all this, contextvars still would not make it into 3.7, I also thought that it might be the right decision. As you may already know, I think there are several problems with this PEP. Would it be worth it to write down some thoughts on this PEP in the morning? -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven +
participants (8)
-
Antoine Pitrou -
Guido van Rossum -
Guido van Rossum -
Koos Zevenhoven -
Nathaniel Smith -
Nick Coghlan -
Victor Stinner -
Yury Selivanov