Mailman 3 August 2017 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists … [View More]now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich [View Less]

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann Feb. 21, 2020

Feb. 21, 2020

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int … [View More]

14 20

Asynchronous exception handling around with/try statement borders
by Erik Bray Sept. 24, 2018

Sept. 24, 2018

Hi folks, I normally wouldn't bring something like this up here, except I think that there is possibility of something to be done--a language documentation clarification if nothing else, though possibly an actual code change as well. I've been having an argument with a colleague over the last couple days over the proper way order of statements when setting up a try/finally to perform cleanup of some action. On some level we're both being stubborn I think, and I'm not looking for resolution … [View More]as to who's right/wrong or I wouldn't bring it to this list in the first place. The original argument was over setting and later restoring os.environ, but we ended up arguing over threading.Lock.acquire/release which I think is a more interesting example of the problem, and he did raise a good point that I do want to bring up. </prologue> My colleague's contention is that given lock = threading.Lock() this is simply *wrong*: lock.acquire() try: do_something() finally: lock.release() whereas this is okay: with lock: do_something() Ignoring other details of how threading.Lock is actually implemented, assuming that Lock.__enter__ calls acquire() and Lock.__exit__ calls release() then as far as I've known ever since Python 2.5 first came out these two examples are semantically *equivalent*, and I can't find any way of reading PEP 343 or the Python language reference that would suggest otherwise. However, there *is* a difference, and has to do with how signals are handled, particularly w.r.t. context managers implemented in C (hence we are talking CPython specifically): If Lock.__enter__ is a pure Python method (even if it maybe calls some C methods), and a SIGINT is handled during execution of that method, then in almost all cases a KeyboardInterrupt exception will be raised from within Lock.__enter__--this means the suite under the with: statement is never evaluated, and Lock.__exit__ is never called. You can be fairly sure the KeyboardInterrupt will be raised from somewhere within a pure Python Lock.__enter__ because there will usually be at least one remaining opcode to be evaluated, such as RETURN_VALUE. Because of how delayed execution of signal handlers is implemented in the pyeval main loop, this means the signal handler for SIGINT will be called *before* RETURN_VALUE, resulting in the KeyboardInterrupt exception being raised. Standard stuff. However, if Lock.__enter__ is a PyCFunction things are quite different. If you look at how the SETUP_WITH opcode is implemented, it first calls the __enter__ method with _PyObjet_CallNoArg. If this returns NULL (i.e. an exception occurred in __enter__) then "goto error" is executed and the exception is raised. However if it returns non-NULL the finally block is set up with PyFrame_BlockSetup and execution proceeds to the next opcode. At this point a potentially waiting SIGINT is handled, resulting in KeyboardInterrupt being raised while inside the with statement's suite, and finally block, and hence Lock.__exit__ are entered. Long story short, because Lock.__enter__ is a C function, assuming that it succeeds normally then with lock: do_something() always guarantees that Lock.__exit__ will be called if a SIGINT was handled inside Lock.__enter__, whereas with lock.acquire() try: ... finally: lock.release() there is at last a small possibility that the SIGINT handler is called after the CALL_FUNCTION op but before the try/finally block is entered (e.g. before executing POP_TOP or SETUP_FINALLY). So the end result is that the lock is held and never released after the KeyboardInterrupt (whether or not it's handled somehow). Whereas, again, if Lock.__enter__ is a pure Python function there's less likely to be any difference (though I don't think the possibility can be ruled out entirely). At the very least I think this quirk of CPython should be mentioned somewhere (since in all other cases the semantic meaning of the "with:" statement is clear). However, I think it might be possible to gain more consistency between these cases if pending signals are checked/handled after any direct call to PyCFunction from within the ceval loop. Sorry for the tl;dr; any thoughts? [View Less]

7 15

Positional-only parameters
by Victor Stinner Sept. 10, 2018

Sept. 10, 2018

Hi, For technical reasons, many functions of the Python standard libraries implemented in C have positional-only parameters. Example: ------- $ ./python Python 3.7.0a0 (default, Feb 25 2017, 04:30:32) >>> help(str.replace) replace(self, old, new, count=-1, /) # <== notice "/" at the end ... >>> "a".replace("x", "y") # ok 'a' >>> "a".replace(old="x", new="y") # ERR! TypeError: replace() takes at least 2 arguments (0 given) ------- When converting the … [View More]

19 27

Implicit string literal concatenation considered harmful?
by Guido van Rossum March 14, 2018

March 14, 2018

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' … [View More]

51 165

Signature Literals
by Blaine Rogers Aug. 30, 2017

Aug. 30, 2017

The current syntax for Callable types is unwieldy, particularly when extended to include varargs and keyword args as in http://mypy.readthedocs.io/en/latest/kinds_of_types.html# extended-callable-types. Why not introduce a signature literal? Proposed syntax: >>> from inspect import Signature, Parameter > >>> () -> > Signature() > >>> (arg0, arg1, arg2=None, arg3=None) -> > Signature( > [Parameter('arg0', Parameter.POSITIONAL_OR_KEYWORD), > … [View More] Parameter('arg1', Parameter.POSITIONAL_OR_KEYWORD), > Parameter('arg2', Parameter.POSITIONAL_OR_KEYWORD, default=None), > Parameter('arg3', Parameter.POSITIONAL_OR_KEYWORD, default=None)], > return_annotation=str > ) > >>> (arg0, arg1: int, arg2=None, arg3: float=None) -> str > Signature( > [Parameter('arg0', Parameter.POSITIONAL_OR_KEYWORD), > Parameter('arg1', Parameter.POSITIONAL_OR_KEYWORD, annotation=int), > Parameter('arg2', Parameter.POSITIONAL_OR_KEYWORD, default=None), > Parameter('arg3', Parameter.POSITIONAL_OR_KEYWORD, annotation=float, > default=None)], > return_annotation=str > ) > >>> (:, :, :, arg1, *, arg2) -> > Signature( > [Parameter('', Parameter.POSITIONAL_ONLY), > Parameter('', Parameter.POSITIONAL_ONLY), > Parameter('', Parameter.POSITIONAL_ONLY), > Parameter('arg1', Parameter.POSITIONAL_OR_KEYWORD), > Parameter('arg2', Parameter.KEYWORD_ONLY)] > ) > >>> (:int, :float, *, keyword: complex) -> str > Signature( > [Parameter('', Parameter.POSITIONAL_ONLY, annotation=int), > Parameter('', Parameter.POSITIONAL_ONLY, annotation=float), > Parameter('keyword', Parameter.KEYWORD_ONLY, annotation=complex)], return_annotation=str ) Compare the above to their equivalents using Callable (and the experimental extension to Mypy): >>> Callable[[], Any] > >>> Callable[[Arg(Any, 'arg0'), Arg(int, 'arg1'), DefaultArg(Any, 'arg2'), > DefaultArg(float, 'kwarg3')], str] > >>> Callable[[Arg(), Arg(), Arg(), Arg(Any, 'arg1'), NamedArg(Any, > 'arg2')], Any] > >>> Callable[[int, float, NamedArg(complex, 'keyword')], Any] The proposed signature literal syntax is shorter, just as clear and imo nicer to read. Here is what it looks like in annotations: from typing import TypeVar, Callable > > A = TypeVar('A') > def apply_successor(func: Callable[[A], A], init: A, n_applications: int) > -> A: ... > def apply_successor(func: (:A) -> A, init: A, n_applications: int) -> A: > ... > > import tensorflow as tf > import numpy as np > > def run(policy: Callable[[np.ndarray, Arg(Dict[tf.Tensor, np.ndarray], > 'updated_feeds')], np.ndarray]) -> bool: ... > def run(policy: (:np.ndarray, updated_feeds: Dict[tf.Tensor, np.ndarray]) > -> np.ndarray) -> bool: ... > # If Mypy accepted literals for container types (dict, set, list, tuple, > etc) this would be nicer still > def run(policy: (:np.ndarray, updated_feeds: {tf.Tensor: np.ndarray}) -> > np.ndarray) -> bool: ... Initial thoughts: - () -> is ugly, but the -> would be necessary to distinguish it from the empty tuple (). Actually, it can be difficult to tell the difference between the proposed signature literals and tuples, especially for long signatures with no annotations or defaults. An alternative would be to prefix the arguments with an @ or other uncommon symbol (maybe &). () -> becomes @(), and it is clear from the start that you're reading a signature. - Supposing the syntax for function definitions was changed to match the proposed signature literals, one could make something like the following possible: >>> def add(:, :): ... arg0, arg1 = __call_signature__.args ... return arg0.value + arg1.value >>> add(1, 2) 3 >>> add('hello', 'world') 'helloworld' Where __call_signature__ is a magic name that evaluates to an inspect.BoundArguments instance representing the signature of the function call. I'm not sure why you'd want functions with positional-only arguments, but now you could have them. - You could further extend the function definition syntax to allow an expression that evaluates to a signature instead of a literal >>> signature = (:, :) -> >>> def add signature: ... arg0, arg1 = __call_signature__.args ... return arg0 + arg1 Again, not sure how useful this would be. [View Less]

4 3

argparse.ArgumentParser: include arguments from files with relative paths
by Robert Schindler Aug. 29, 2017

Aug. 29, 2017

Hello, I'm often using ArgumentParser in my projects, as well as its ability to read argument lists from files. However, the problem is that nested includes of such argument files have to specify paths relative to os.getcwd(), no matter where the file containing the include statement is located. Currently, this can be circumvented by always using absolute paths. But imho that is not a practical solution, due to the obvious portability issues it causes. I suggest adding a new parameter to … [View More]

1 0

Unittest error message failure context lazy creation
by francismb Aug. 27, 2017

Aug. 27, 2017

Hi all, while using `unittest` I see the pattern of creating an error message with the test context for the case that some `assert...` methods fails (to get a good error message). On the lines: class Test...(unittest.TestCase): longMessage = True def test_(self): ... for a, b, c ... in zip(A, B, C, ..): * call the function under test and get the result msg = "Some headline: {}{} ...".format(a, b, c,..) self.assert...( ,msg) The `msg` is just used in case the … [View More]

3 3

Re: [Python-ideas] Please consider adding context manager versions of setUp/tearDown to unittest.TestCase
by rymg19＠gmail.com Aug. 24, 2017

Aug. 24, 2017

TBH you're completely right. Every time I see someone using unittest andItsHorriblyUnpythonicNames, I want to kill a camel. Sometimes, though, I feel like part of the struggle is the alternative. If you dislike unittest, but pytest is too "magical" for you, what do you use? Many Python testing tools like nose are just test *runners*, so you still need something else. In the end, many just end up back at unittest, maybe with nose on top. As much as I hate JavaScript, their testing libraries … [View More]are leagues above what Python has. -- Ryan (ライアン) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone elsehttp://refi64.com On Aug 22, 2017 at 5:09 PM, <Chris Barker <chris.barker(a)noaa.gov>> wrote: ** Caution: cranky curmudgeonly opinionated comment ahead: ** unitest is such an ugly Java-esque static mess of an API that there's really no point in trying to clean it up and make it more pythonic -- go off and use pytest and be happier. -CHB On Tue, Aug 22, 2017 at 5:42 AM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > On 22 August 2017 at 15:34, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > > On 21 August 2017 at 11:32, Neil Girdhar <mistersheik(a)gmail.com> wrote: > >> This question describes an example of the problem: > >> https://stackoverflow.com/questions/8416208/in-python- > is-there-a-good-idiom-for-using-context-managers-in-setup-teardown. > >> You want to invoke a context manager in your setup/tearing-down, but the > >> easiest way to do that is to override run, which seems ugly. > > > > Using context managers when you can't use a with statement is one of > > the main use cases for contextlib.ExitStack(): > > > > def setUp(self): > > self._resource_stack = stack = contextlib.ExitStack() > > self._resource = stack.enter_context(MyResource()) > > > > def tearDown(self): > > self._resource_stack.close() > > > > I posted that as an additional answer to the question: > > https://stackoverflow.com/questions/8416208/in-python- > is-there-a-good-idiom-for-using-context-managers-in- > setup-teardown/45809502#45809502 > > Sjoerd pointed out off-list that this doesn't cover the case where > you're acquiring multiple resources and one of the later acquisitions > fails, so I added the ExitStack idiom that covers that case (using > stack.pop_all() as the last operation in a with statement): > > def setUp(self): > with contextlib.ExitStack() as stack: > self._resource1 = stack.enter_context(GetResource()) > self._resource2 = stack.enter_context(GetOtherResource()) > # Failures before here -> immediate cleanup > self.addCleanup(stack.pop_all().close) > # Now cleanup won't happen until the cleanup functions run > > I also remember that using addCleanup lets you avoid defining tearDown > entirely. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ [View Less]

7 11

PEP 550 dumbed down
by Jim J. Jewett Aug. 24, 2017

Aug. 24, 2017

I know I'm not the only one who is confused by at least some of the alternative terminology choices. I suspect I'm not the only one who sometimes missed part of the argument because I was distracted figuring out what the objects were, and forgot to verify what was being done and why. I also suspect that it could be much simpler to follow if the API were designed in the abstract, with the implementation left for later. So is the following API missing anything important? (1) Get the current (… [View More]

5 4

Re: [Python-ideas] Remote package/module imports through HTTP/S
by John Torakis Aug. 24, 2017

Aug. 24, 2017

On 23/08/2017 20:36, John Torakis wrote: > Yeah, I am a security researcher, I am keen on backdoor programming and > staging and all that! It is my official job and research topic! I go to > the office and code such stuff! I am not a blackhat, nor a security > enthusiast, it is my job. > > > First of all, let's all agree that if someone can run Python code in > your computer you are 100% hacked! It is irrelevant if "httpimport" is a > core python feature or not in … [View More]that case. > > Now, I agree that this can be exploited if used under plain HTTP, it is > a MiTM -> Remote code execution case. I admit that this is not bright. > But I mention that this can be used in testing. > > On the topic of HTTPS, man-in-the-middle is not possible without > previous Trusted Certificate compromise. Github can be trusted 100% > percent for example. A certificate check has to take place in the HTTPS > remote loading for sure! > > When I said a "core feature" I meant that the "httpimport" module would > deliver with the core modules. Not that the Finder/Loader has to be in > the list of Finders/Loaders that are used by default! For god sake, I > wouldn't like my PC to start probing for modules just because I mistyped > an import line! > > I know that pip works nicely, especially when paired with virtual > environments, but ad-hoc importing is another another thing. It isn't > meant for delivering real projects. Just for testing modules without the > need to download them, maybe install them, and all. > > > Thank you for your time, > John Torakis > > > On 23/08/2017 20:17, Chris Angelico wrote: >> On Thu, Aug 24, 2017 at 2:55 AM, John Torakis <john.torakis(a)gmail.com> wrote: >>> Hello all! >>> >>> Today I opened an issue in bugs.python.org >>> (http://bugs.python.org/issue31264) proposing a module I created for >>> remote package/module imports through standard HTTP/S. >>> >>> The concept is that, if a directory is served through HTTP/S (the way >>> SimpleHTTPServer module serves directories), a Finder/Loader object can >>> fetch Python files from that directory using HTTP requests, and finally >>> load them as modules (or packages) in the running namespace. >>> >>> The repo containing a primitive (but working) version of the >>> Finder/Loader, also contains self explanatory examples (in the README.md): >>> >>> https://github.com/operatorequals/httpimport >>> >>> >>> My proposal is that this module can become a core Python feature, >>> providing a way to load modules even from Github.com repositories, >>> without the need to "git clone - setup.py install" them. >>> >>> >>> Other languages, like golang, provide this functionality from their >>> early days (day one?). Python development can be greatly improved if a >>> "try before pip installing" mechanism gets in place, as it will add a >>> lot to the REPL nature of the testing/experimenting process. >> As a core feature? No no no no no no no no. Absolutely do NOT WANT >> THIS. This is a security bug magnet; can you imagine trying to ensure >> that malicious code is not executed, in an arbitrary execution >> context? As an explicitly-enabled feature, it's a lot less hairy than >> a permanently-active one (can you IMAGINE how terrifying that would >> be?), but even so, trying to prove that addRemoteRepo (not a >> PEP8-compliant name, btw) is getting the correct code is not going to >> be easy. You have to (a) drop HTTP altogether and mandate SSL and (b) >> be absolutely sure that your certificate chains are 100% dependable, >> which - as we've seen recently - is a nontrivial task. >> >> The easiest way to add remote code is pip. For most packages, that's >> what you want to be using: >> >> pip install requests >> >> will make "import requests" functional. I don't see pip mentioned >> anywhere in your README, but you do mention the testing of pull >> requests, so at very least, this wants some explanatory screed. >> >> But I'm not entirely sure I want to support this. You're explicitly >> talking about using this with the creation of backdoors... in what, >> exactly? What are you actually getting at here? >> >> ChrisA >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas(a)python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > [View Less]

7 19

PEP 550 v2
by Yury Selivanov Aug. 23, 2017

Aug. 23, 2017

Hi, Here's the PEP 550 version 2. Thanks to a very active and insightful discussion here on Python-ideas, we've discovered a number of problems with the first version of the PEP. This version is a complete rewrite (only Abstract, Rationale, and Goals sections were not updated). The updated PEP is live on python.org: https://www.python.org/dev/peps/pep-0550/ There is no reference implementation at this point, but I'm confident that this version of the spec will have the same extremely low … [View More]runtime overhead as the first version. Thanks to the new ContextItem design, accessing values in the context is even faster now. Thank you! PEP: 550 Title: Execution Context Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov <yury(a)magic.io> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-Aug-2017 Python-Version: 3.7 Post-History: 11-Aug-2017, 15-Aug-2017 Abstract ======== This PEP proposes a new mechanism to manage execution state--the logical environment in which a function, a thread, a generator, or a coroutine executes in. A few examples of where having a reliable state storage is required: * Context managers like decimal contexts, ``numpy.errstate``, and ``warnings.catch_warnings``; * Storing request-related data such as security tokens and request data in web applications, implementing i18n; * Profiling, tracing, and logging in complex and large code bases. The usual solution for storing state is to use a Thread-local Storage (TLS), implemented in the standard library as ``threading.local()``. Unfortunately, TLS does not work for the purpose of state isolation for generators or asynchronous code, because such code executes concurrently in a single thread. Rationale ========= Traditionally, a Thread-local Storage (TLS) is used for storing the state. However, the major flaw of using the TLS is that it works only for multi-threaded code. It is not possible to reliably contain the state within a generator or a coroutine. For example, consider the following generator:: def calculate(precision, ...): with decimal.localcontext() as ctx: # Set the precision for decimal calculations # inside this block ctx.prec = precision yield calculate_something() yield calculate_something_else() Decimal context is using a TLS to store the state, and because TLS is not aware of generators, the state can leak. If a user iterates over the ``calculate()`` generator with different precisions one by one using a ``zip()`` built-in, the above code will not work correctly. For example:: g1 = calculate(precision=100) g2 = calculate(precision=50) items = list(zip(g1, g2)) # items[0] will be a tuple of: # first value from g1 calculated with 100 precision, # first value from g2 calculated with 50 precision. # # items[1] will be a tuple of: # second value from g1 calculated with 50 precision (!!!), # second value from g2 calculated with 50 precision. An even scarier example would be using decimals to represent money in an async/await application: decimal calculations can suddenly lose precision in the middle of processing a request. Currently, bugs like this are extremely hard to find and fix. Another common need for web applications is to have access to the current request object, or security context, or, simply, the request URL for logging or submitting performance tracing data:: async def handle_http_request(request): context.current_http_request = request await ... # Invoke your framework code, render templates, # make DB queries, etc, and use the global # 'current_http_request' in that code. # This isn't currently possible to do reliably # in asyncio out of the box. These examples are just a few out of many, where a reliable way to store context data is absolutely needed. The inability to use TLS for asynchronous code has lead to proliferation of ad-hoc solutions, which are limited in scope and do not support all required use cases. Current status quo is that any library, including the standard library, that uses a TLS, will likely not work as expected in asynchronous code or with generators (see [3]_ as an example issue.) Some languages that have coroutines or generators recommend to manually pass a ``context`` object to every function, see [1]_ describing the pattern for Go. This approach, however, has limited use for Python, where we have a huge ecosystem that was built to work with a TLS-like context. Moreover, passing the context explicitly does not work at all for libraries like ``decimal`` or ``numpy``, which use operator overloading. .NET runtime, which has support for async/await, has a generic solution of this problem, called ``ExecutionContext`` (see [2]_). On the surface, working with it is very similar to working with a TLS, but the former explicitly supports asynchronous code. Goals ===== The goal of this PEP is to provide a more reliable alternative to ``threading.local()``. It should be explicitly designed to work with Python execution model, equally supporting threads, generators, and coroutines. An acceptable solution for Python should meet the following requirements: * Transparent support for code executing in threads, coroutines, and generators with an easy to use API. * Negligible impact on the performance of the existing code or the code that will be using the new mechanism. * Fast C API for packages like ``decimal`` and ``numpy``. Explicit is still better than implicit, hence the new APIs should only be used when there is no acceptable way of passing the state explicitly. Specification ============= Execution Context is a mechanism of storing and accessing data specific to a logical thread of execution. We consider OS threads, generators, and chains of coroutines (such as ``asyncio.Task``) to be variants of a logical thread. In this specification, we will use the following terminology: * **Local Context**, or LC, is a key/value mapping that stores the context of a logical thread. * **Execution Context**, or EC, is an OS-thread-specific dynamic stack of Local Contexts. * **Context Item**, or CI, is an object used to set and get values from the Execution Context. Please note that throughout the specification we use simple pseudo-code to illustrate how the EC machinery works. The actual algorithms and data structures that we will use to implement the PEP are discussed in the `Implementation Strategy`_ section. Context Item Object ------------------- The ``sys.new_context_item(description)`` function creates a new ``ContextItem`` object. The ``description`` parameter is a ``str``, explaining the nature of the context key for introspection and debugging purposes. ``ContextItem`` objects have the following methods and attributes: * ``.description``: read-only description; * ``.set(o)`` method: set the value to ``o`` for the context item in the execution context. * ``.get()`` method: return the current EC value for the context item. Context items are initialized with ``None`` when created, so this method call never fails. The below is an example of how context items can be used:: my_context = sys.new_context_item(description='mylib.context') my_context.set('spam') # Later, to access the value of my_context: print(my_context.get()) Thread State and Multi-threaded code ------------------------------------ Execution Context is implemented on top of Thread-local Storage. For every thread there is a separate stack of Local Contexts -- mappings of ``ContextItem`` objects to their values in the LC. New threads always start with an empty EC. For CPython:: PyThreadState: execution_context: ExecutionContext([ LocalContext({ci1: val1, ci2: val2, ...}), ... ]) The ``ContextItem.get()`` and ``.set()`` methods are defined as follows (in pseudo-code):: class ContextItem: def get(self): tstate = PyThreadState_Get() for local_context in reversed(tstate.execution_context): if self in local_context: return local_context[self] def set(self, value): tstate = PyThreadState_Get() if not tstate.execution_context: tstate.execution_context = [LocalContext()] tstate.execution_context[-1][self] = value With the semantics defined so far, the Execution Context can already be used as an alternative to ``threading.local()``:: def print_foo(): print(ci.get() or 'nothing') ci = sys.new_context_item(description='test') ci.set('foo') # Will print "foo": print_foo() # Will print "nothing": threading.Thread(target=print_foo).start() Manual Context Management ------------------------- Execution Context is generally managed by the Python interpreter, but sometimes it is desirable for the user to take the control over it. A few examples when this is needed: * running a computation in ``concurrent.futures.ThreadPoolExecutor`` with the current EC; * reimplementing generators with iterators (more on that later); * managing contexts in asynchronous frameworks (implement proper EC support in ``asyncio.Task`` and ``asyncio.loop.call_soon``.) For these purposes we add a set of new APIs (they will be used in later sections of this specification): * ``sys.new_local_context()``: create an empty ``LocalContext`` object. * ``sys.new_execution_context()``: create an empty ``ExecutionContext`` object. * Both ``LocalContext`` and ``ExecutionContext`` objects are opaque to Python code, and there are no APIs to modify them. * ``sys.get_execution_context()`` function. The function returns a copy of the current EC: an ``ExecutionContext`` instance. The runtime complexity of the actual implementation of this function can be O(1), but for the purposes of this section it is equivalent to:: def get_execution_context(): tstate = PyThreadState_Get() return copy(tstate.execution_context) * ``sys.run_with_execution_context(ec: ExecutionContext, func, *args, **kwargs)`` runs ``func(*args, **kwargs)`` in the provided execution context:: def run_with_execution_context(ec, func, *args, **kwargs): tstate = PyThreadState_Get() old_ec = tstate.execution_context tstate.execution_context = ExecutionContext( ec.local_contexts + [LocalContext()] ) try: return func(*args, **kwargs) finally: tstate.execution_context = old_ec Any changes to Local Context by ``func`` will be ignored. This allows to reuse one ``ExecutionContext`` object for multiple invocations of different functions, without them being able to affect each other's environment:: ci = sys.new_context_item('example') ci.set('spam') def func(): print(ci.get()) ci.set('ham') ec = sys.get_execution_context() sys.run_with_execution_context(ec, func) sys.run_with_execution_context(ec, func) # Will print: # spam # spam * ``sys.run_with_local_context(lc: LocalContext, func, *args, **kwargs)`` runs ``func(*args, **kwargs)`` in the current execution context using the specified local context. Any changes that ``func`` does to the local context will be persisted in ``lc``. This behaviour is different from the ``run_with_execution_context()`` function, which always creates a new throw-away local context. In pseudo-code:: def run_with_local_context(lc, func, *args, **kwargs): tstate = PyThreadState_Get() old_ec = tstate.execution_context tstate.execution_context = ExecutionContext( old_ec.local_contexts + [lc] ) try: return func(*args, **kwargs) finally: tstate.execution_context = old_ec Using the previous example:: ci = sys.new_context_item('example') ci.set('spam') def func(): print(ci.get()) ci.set('ham') ec = sys.get_execution_context() lc = sys.new_local_context() sys.run_with_local_context(lc, func) sys.run_with_local_context(lc, func) # Will print: # spam # ham As an example, let's make a subclass of ``concurrent.futures.ThreadPoolExecutor`` that preserves the execution context for scheduled functions:: class Executor(concurrent.futures.ThreadPoolExecutor): def submit(self, fn, *args, **kwargs): context = sys.get_execution_context() fn = functools.partial( sys.run_with_execution_context, context, fn, *args, **kwargs) return super().submit(fn) EC Semantics for Coroutines --------------------------- Python :pep:`492` coroutines are used to implement cooperative multitasking. For a Python end-user they are similar to threads, especially when it comes to sharing resources or modifying the global state. An event loop is needed to schedule coroutines. Coroutines that are explicitly scheduled by the user are usually called Tasks. When a coroutine is scheduled, it can schedule other coroutines using an ``await`` expression. In async/await world, awaiting a coroutine is equivalent to a regular function call in synchronous code. Thus, Tasks are similar to threads. By drawing a parallel between regular multithreaded code and async/await, it becomes apparent that any modification of the execution context within one Task should be visible to all coroutines scheduled within it. Any execution context modifications, however, must not be visible to other Tasks executing within the same OS thread. Coroutine Object Modifications ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To achieve this, a small set of modifications to the coroutine object is needed: * New ``cr_local_context`` attribute. This attribute is readable and writable for Python code. * When a coroutine object is instantiated, its ``cr_local_context`` is initialized with an empty Local Context. * Coroutine's ``.send()`` and ``.throw()`` methods are modified as follows (in pseudo-C):: if coro.cr_local_context is not None: tstate = PyThreadState_Get() tstate.execution_context.push(coro.cr_local_context) try: # Perform the actual `Coroutine.send()` or # `Coroutine.throw()` call. return coro.send(...) finally: coro.cr_local_context = tstate.execution_context.pop() else: # Perform the actual `Coroutine.send()` or # `Coroutine.throw()` call. return coro.send(...) * When Python interpreter sees an ``await`` instruction, it inspects the ``cr_local_context`` attribute of the coroutine that is about to be awaited. For ``await coro``: * If ``coro.cr_local_context`` is an empty ``LocalContext`` object that ``coro`` was created with, the interpreter will set ``coro.cr_local_context`` to ``None``. * If ``coro.cr_local_context`` was modified by Python code, the interpreter will leave it as is. This makes any changes to execution context made by nested coroutine calls within a Task to be visible throughout the Task:: ci = sys.new_context_item('example') async def nested(): ci.set('nested') asynd def main(): ci.set('main') print('before:', ci.get()) await nested() print('after:', ci.get()) # Will print: # before: main # after: nested Essentially, coroutines work with Execution Context items similarly to threads, and ``await`` expression acts like a function call. This mechanism also works for ``yield from`` in generators decorated with ``(a)types.coroutine`` or ``(a)asyncio.coroutine``, which are called "generator-based coroutines" according to :pep:`492`, and should be fully compatible with native async/await coroutines. Tasks ^^^^^ In asynchronous frameworks like asyncio, coroutines are run by an event loop, and need to be explicitly scheduled (in asyncio coroutines are run by ``asyncio.Task``.) With the currently defined semantics, the interpreter makes coroutines linked by an ``await`` expression share the same Local Context. The interpreter, however, is not aware of the Task concept, and cannot help with ensuring that new Tasks started in coroutines, use the correct EC:: current_request = sys.new_context_item(description='request') async def child(): print('current request:', repr(current_request.get())) async def handle_request(request): current_request.set(request) event_loop.create_task(child) run(top_coro()) # Will print: # current_request: None To enable correct Execution Context propagation into Tasks, the asynchronous framework needs to assist the interpreter: * When ``create_task`` is called, it should capture the current execution context with ``sys.get_execution_context()`` and save it on the Task object. * When the Task object runs its coroutine object, it should execute ``.send()`` and ``.throw()`` methods within the captured execution context, using the ``sys.run_with_execution_context()`` function. With help from the asynchronous framework, the above snippet will run correctly, and the ``child()`` coroutine will be able to access the current request object through the ``current_request`` Context Item. Event Loop Callbacks ^^^^^^^^^^^^^^^^^^^^ Similarly to Tasks, functions like asyncio's ``loop.call_soon()`` should capture the current execution context with ``sys.get_execution_context()`` and execute callbacks within it with ``sys.run_with_execution_context()``. This way the following code will work:: current_request = sys.new_context_item(description='request') def log(): request = current_request.get() print(request) async def request_handler(request): current_request.set(request) get_event_loop.call_soon(log) Generators ---------- Generators in Python, while similar to Coroutines, are used in a fundamentally different way. They are producers of data, and they use ``yield`` expression to suspend/resume their execution. A crucial difference between ``await coro`` and ``yield value`` is that the former expression guarantees that the ``coro`` will be executed fully, while the latter is producing ``value`` and suspending the generator until it gets iterated again. Generators, similarly to coroutines, have a ``gi_local_context`` attribute, which is set to an empty Local Context when created. Contrary to coroutines though, ``yield from o`` expression in generators (that are not generator-based coroutines) is semantically equivalent to ``for v in o: yield v``, therefore the interpreter does not attempt to control their ``gi_local_context``. EC Semantics for Generators ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Every generator object has its own Local Context that stores only its own local modifications of the context. When a generator is being iterated, its local context will be put in the EC stack of the current thread. This means that the generator will be able to see access items from the surrounding context:: local = sys.new_context_item("local") global = sys.new_context_item("global") def generator(): local.set('inside gen:') while True: print(local.get(), global.get()) yield g = gen() local.set('hello') global.set('spam') next(g) local.set('world') global.set('ham') next(g) # Will print: # inside gen: spam # inside gen: ham Any changes to the EC in nested generators are invisible to the outer generator:: local = sys.new_context_item("local") def inner_gen(): local.set('spam') yield def outer_gen(): local.set('ham') yield from gen() print(local.get()) list(outer_gen()) # Will print: # ham Running generators without LC ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Similarly to coroutines, generators with ``gi_local_context`` set to ``None`` simply use the outer Local Context. The ``(a)contextlib.contextmanager`` decorator uses this mechanism to allow its generator to affect the EC:: item = sys.new_context_item('test') @contextmanager def context(x): old = item.get() item.set('x') try: yield finally: item.set(old) with context('spam'): with context('ham'): print(1, item.get()) print(2, item.get()) # Will print: # 1 ham # 2 spam Implementing Generators with Iterators ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The Execution Context API allows to fully replicate EC behaviour imposed on generators with a regular Python iterator class:: class Gen: def __init__(self): self.local_context = sys.new_local_context() def __iter__(self): return self def __next__(self): return sys.run_with_local_context( self.local_context, self._next_impl) def _next_impl(self): # Actual __next__ implementation. ... Asynchronous Generators ----------------------- Asynchronous Generators (AG) interact with the Execution Context similarly to regular generators. They have an ``ag_local_context`` attribute, which, similarly to regular generators, can be set to ``None`` to make them use the outer Local Context. This is used by the new ``contextlib.asynccontextmanager`` decorator. The EC support of ``await`` expression is implemented using the same approach as in coroutines, see the `Coroutine Object Modifications`_ section. Greenlets --------- Greenlet is an alternative implementation of cooperative scheduling for Python. Although greenlet package is not part of CPython, popular frameworks like gevent rely on it, and it is important that greenlet can be modified to support execution contexts. In a nutshell, greenlet design is very similar to design of generators. The main difference is that for generators, the stack is managed by the Python interpreter. Greenlet works outside of the Python interpreter, and manually saves some ``PyThreadState`` fields and pushes/pops the C-stack. Thus the ``greenlet`` package can be easily updated to use the new low-level `C API`_ to enable full support of EC. New APIs ======== Python ------ Python APIs were designed to completely hide the internal implementation details, but at the same time provide enough control over EC and LC to re-implement all of Python built-in objects in pure Python. 1. ``sys.new_context_item(description='...')``: create a ``ContextItem`` object used to access/set values in EC. 2. ``ContextItem``: * ``.description``: read-only attribute. * ``.get()``: return the current value for the item. * ``.set(o)``: set the current value in the EC for the item. 3. ``sys.get_execution_context()``: return the current ``ExecutionContext``. 4. ``sys.new_execution_context()``: create a new empty ``ExecutionContext``. 5. ``sys.new_local_context()``: create a new empty ``LocalContext``. 6. ``sys.run_with_execution_context(ec: ExecutionContext, func, *args, **kwargs)``. 7. ``sys.run_with_local_context(lc:LocalContext, func, *args, **kwargs)``. C API ----- 1. ``PyContextItem * PyContext_NewItem(char *desc)``: create a ``PyContextItem`` object. 2. ``PyObject * PyContext_GetItem(PyContextItem *)``: get the current value for the context item. 3. ``int PyContext_SetItem(PyContextItem *, PyObject *)``: set the current value for the context item. 4. ``PyLocalContext * PyLocalContext_New()``: create a new empty ``PyLocalContext``. 5. ``PyLocalContext * PyExecutionContext_New()``: create a new empty ``PyExecutionContext``. 6. ``PyExecutionContext * PyExecutionContext_Get()``: get the EC for the active thread state. 7. ``int PyExecutionContext_Set(PyExecutionContext *)``: set the passed EC object as the current for the active thread state. 8. ``int PyExecutionContext_SetWithLocalContext(PyExecutionContext *, PyLocalContext *)``: allows to implement ``sys.run_with_local_context`` Python API. Implementation Strategy ======================= LocalContext is a Weak Key Mapping ---------------------------------- Using a weak key mapping for ``LocalContext`` implementation enables the following properties with regards to garbage collection: * ``ContextItem`` objects are strongly-referenced only from the application code, not from any of the Execution Context machinery or values they point to. This means that there are no reference cycles that could extend their lifespan longer than necessary, or prevent their garbage collection. * Values put in the Execution Context are guaranteed to be kept alive while there is a ``ContextItem`` key referencing them in the thread. * If a ``ContextItem`` is garbage collected, all of its values will be removed from all contexts, allowing them to be GCed if needed. * If a thread has ended its execution, its thread state will be cleaned up along with its ``ExecutionContext``, cleaning up all values bound to all Context Items in the thread. ContextItem.get() Cache ----------------------- We can add three new fields to ``PyThreadState`` and ``PyInterpreterState`` structs: * ``uint64_t PyThreadState->unique_id``: a globally unique thread state identifier (we can add a counter to ``PyInterpreterState`` and increment it when a new thread state is created.) * ``uint64_t PyInterpreterState->context_item_deallocs``: every time a ``ContextItem`` is GCed, all Execution Contexts in all threads will lose track of it. ``context_item_deallocs`` will simply count all ``ContextItem`` deallocations. * ``uint64_t PyThreadState->execution_context_ver``: every time a new item is set, or an existing item is updated, or the stack of execution contexts is changed in the thread, we increment this counter. The above two fields allow implementing a fast cache path in ``ContextItem.get()``, in pseudo-code:: class ContextItem: def get(self): tstate = PyThreadState_Get() if (self.last_tstate_id == tstate.unique_id and self.last_ver == tstate.execution_context_ver self.last_deallocs == tstate.iterp.context_item_deallocs): return self.last_value value = None for mapping in reversed(tstate.execution_context): if self in mapping: value = mapping[self] break self.last_value = value self.last_tstate_id = tstate.unique_id self.last_ver = tstate.execution_context_ver self.last_deallocs = tstate.interp.context_item_deallocs return value This is similar to the trick that decimal C implementation uses for caching the current decimal context, and will have the same performance characteristics, but available to all Execution Context users. Approach #1: Use a dict for LocalContext ---------------------------------------- The straightforward way of implementing the proposed EC mechanisms is to create a ``WeakKeyDict`` on top of Python ``dict`` type. To implement the ``ExecutionContext`` type we can use Python ``list`` (or a custom stack implementation with some pre-allocation optimizations). This approach will have the following runtime complexity: * O(M) for ``ContextItem.get()``, where ``M`` is the number of Local Contexts in the stack. It is important to note that ``ContextItem.get()`` will implement a cache making the operation O(1) for packages like ``decimal`` and ``numpy``. * O(1) for ``ContextItem.set()``. * O(N) for ``sys.get_execution_context()``, where ``N`` is the total number of items in the current **execution** context. Approach #2: Use HAMT for LocalContext -------------------------------------- Languages like Clojure and Scala use Hash Array Mapped Tries (HAMT) to implement high performance immutable collections [5]_, [6]_. Immutable mappings implemented with HAMT have O(log\ :sub:`32`\ N) performance for both ``set()``, ``get()``, and ``merge()`` operations, which is essentially O(1) for relatively small mappings (read about HAMT performance in CPython in the `Appendix: HAMT Performance`_ section.) In this approach we use the same design of the ``ExecutionContext`` as in Approach #1, but we will use HAMT backed weak key Local Context implementation. With that we will have the following runtime complexity: * O(M * log\ :sub:`32`\ N) for ``ContextItem.get()``, where ``M`` is the number of Local Contexts in the stack, and ``N`` is the number of items in the EC. The operation will essentially be O(M), because execution contexts are normally not expected to have more than a few dozen of items. (``ContextItem.get()`` will have the same caching mechanism as in Approach #1.) * O(log\ :sub:`32`\ N) for ``ContextItem.set()`` where ``N`` is the number of items in the current **local** context. This will essentially be an O(1) operation most of the time. * O(log\ :sub:`32`\ N) for ``sys.get_execution_context()``, where ``N`` is the total number of items in the current **execution** context. Essentially, using HAMT for Local Contexts instead of Python dicts, allows to bring down the complexity of ``sys.get_execution_context()`` from O(N) to O(log\ :sub:`32`\ N) because of the more efficient merge algorithm. Approach #3: Use HAMT and Immutable Linked List ----------------------------------------------- We can make an alternative ``ExecutionContext`` design by using a linked list. Each ``LocalContext`` in the ``ExecutionContext`` object will be wrapped in a linked-list node. ``LocalContext`` objects will use an HAMT backed weak key implementation described in the Approach #2. Every modification to the current ``LocalContext`` will produce a new version of it, which will be wrapped in a **new linked list node**. Essentially this means, that ``ExecutionContext`` is an immutable forest of ``LocalContext`` objects, and can be safely copied by reference in ``sys.get_execution_context()`` (eliminating the expensive "merge" operation.) With this approach, ``sys.get_execution_context()`` will be an **O(1) operation**. Summary ------- We believe that approach #3 enables an efficient and complete Execution Context implementation, with excellent runtime performance. `ContextItem.get() Cache`_ enables fast retrieval of context items for performance critical libraries like decimal and numpy. Fast ``sys.get_execution_context()`` enables efficient management of execution contexts in asynchronous libraries like asyncio. Design Considerations ===================== Can we fix ``PyThreadState_GetDict()``? --------------------------------------- ``PyThreadState_GetDict`` is a TLS, and some of its existing users might depend on it being just a TLS. Changing its behaviour to follow the Execution Context semantics would break backwards compatibility. PEP 521 ------- :pep:`521` proposes an alternative solution to the problem: enhance Context Manager Protocol with two new methods: ``__suspend__`` and ``__resume__``. To make it compatible with async/await, the Asynchronous Context Manager Protocol will also need to be extended with ``__asuspend__`` and ``__aresume__``. This allows to implement context managers like decimal context and ``numpy.errstate`` for generators and coroutines. The following code:: class Context: def __enter__(self): self.old_x = get_execution_context_item('x') set_execution_context_item('x', 'something') def __exit__(self, *err): set_execution_context_item('x', self.old_x) would become this:: local = threading.local() class Context: def __enter__(self): self.old_x = getattr(local, 'x', None) local.x = 'something' def __suspend__(self): local.x = self.old_x def __resume__(self): local.x = 'something' def __exit__(self, *err): local.x = self.old_x Besides complicating the protocol, the implementation will likely negatively impact performance of coroutines, generators, and any code that uses context managers, and will notably complicate the interpreter implementation. :pep:`521` also does not provide any mechanism to propagate state in a local context, like storing a request object in an HTTP request handler to have better logging. Nor does it solve the leaking state problem for greenlet/gevent. Can Execution Context be implemented outside of CPython? -------------------------------------------------------- Because async/await code needs an event loop to run it, an EC-like solution can be implemented in a limited way for coroutines. Generators, on the other hand, do not have an event loop or trampoline, making it impossible to intercept their ``yield`` points outside of the Python interpreter. Backwards Compatibility ======================= This proposal preserves 100% backwards compatibility. Appendix: HAMT Performance ========================== To assess if HAMT can be used for Execution Context, we implemented it in CPython [7]_. .. figure:: pep-0550-hamt_vs_dict.png :align: center :width: 100% Figure 1. Benchmark code can be found here: [9]_. Figure 1 shows that HAMT indeed displays O(1) performance for all benchmarked dictionary sizes. For dictionaries with less than 100 items, HAMT is a bit slower than Python dict/shallow copy. .. figure:: pep-0550-lookup_hamt.png :align: center :width: 100% Figure 2. Benchmark code can be found here: [10]_. Figure 2 shows comparison of lookup costs between Python dict and an HAMT immutable mapping. HAMT lookup time is 30-40% worse than Python dict lookups on average, which is a very good result, considering how well Python dicts are optimized. Note, that according to [8]_, HAMT design can be further improved. Acknowledgments =============== I thank Elvis Pranskevichus and Victor Petrovykh for countless discussions around the topic and PEP proof reading and edits. Thanks to Nathaniel Smith for proposing the ``ContextItem`` design [17]_ [18]_, for pushing the PEP towards a more complete design, and coming up with the idea of having a stack of contexts in the thread state. Thanks to Nick Coghlan for numerous suggestions and ideas on the mailing list, and for coming up with a case that cause the complete rewrite of the initial PEP version [19]_. References ========== .. [1] https://blog.golang.org/context .. [2] https://msdn.microsoft.com/en-us/library/system.threading.executioncontext.… .. [3] https://github.com/numpy/numpy/issues/9444 .. [4] http://bugs.python.org/issue31179 .. [5] https://en.wikipedia.org/wiki/Hash_array_mapped_trie .. [6] http://blog.higher-order.net/2010/08/16/assoc-and-clojures-persistenthashma… .. [7] https://github.com/1st1/cpython/tree/hamt .. [8] https://michael.steindorfer.name/publications/oopsla15.pdf .. [9] https://gist.github.com/1st1/9004813d5576c96529527d44c5457dcd .. [10] https://gist.github.com/1st1/dbe27f2e14c30cce6f0b5fddfc8c437e .. [11] https://github.com/1st1/cpython/tree/pep550 .. [12] https://www.python.org/dev/peps/pep-0492/#async-await .. [13] https://github.com/MagicStack/uvloop/blob/master/examples/bench/echoserver.… .. [14] https://github.com/MagicStack/pgbench .. [15] https://github.com/python/performance .. [16] https://gist.github.com/1st1/6b7a614643f91ead3edf37c4451a6b4c .. [17] https://mail.python.org/pipermail/python-ideas/2017-August/046752.html .. [18] https://mail.python.org/pipermail/python-ideas/2017-August/046772.html .. [19] https://mail.python.org/pipermail/python-ideas/2017-August/046780.html Copyright ========= This document has been placed in the public domain. [View Less]

11 45

Re: [Python-ideas] Remote package/module imports through HTTP/S
by John Torakis Aug. 23, 2017

Aug. 23, 2017

Bounced back on list -------- Forwarded Message -------- Θέμα: Re: [Python-ideas] Remote package/module imports through HTTP/S Ημερομηνία: Wed, 23 Aug 2017 20:36:19 +0300 Από: John Torakis <john.torakis(a)gmail.com> Προς: Chris Angelico <rosuav(a)gmail.com> Yeah, I am a security researcher, I am keen on backdoor programming and staging and all that! It is my official job and research topic! I go to the office and code such stuff! I am not a blackhat, nor a security … [View More]enthusiast, it is my job. First of all, let's all agree that if someone can run Python code in your computer you are 100% hacked! It is irrelevant if "httpimport" is a core python feature or not in that case. Now, I agree that this can be exploited if used under plain HTTP, it is a MiTM -> Remote code execution case. I admit that this is not bright. But I mention that this can be used in testing. On the topic of HTTPS, man-in-the-middle is not possible without previous Trusted Certificate compromise. Github can be trusted 100% percent for example. A certificate check has to take place in the HTTPS remote loading for sure! When I said a "core feature" I meant that the "httpimport" module would deliver with the core modules. Not that the Finder/Loader has to be in the list of Finders/Loaders that are used by default! For god sake, I wouldn't like my PC to start probing for modules just because I mistyped an import line! I know that pip works nicely, especially when paired with virtual environments, but ad-hoc importing is another another thing. It isn't meant for delivering real projects. Just for testing modules without the need to download them, maybe install them, and all. Thank you for your time, John Torakis On 23/08/2017 20:17, Chris Angelico wrote: > On Thu, Aug 24, 2017 at 2:55 AM, John Torakis <john.torakis(a)gmail.com> wrote: >> Hello all! >> >> Today I opened an issue in bugs.python.org >> (http://bugs.python.org/issue31264) proposing a module I created for >> remote package/module imports through standard HTTP/S. >> >> The concept is that, if a directory is served through HTTP/S (the way >> SimpleHTTPServer module serves directories), a Finder/Loader object can >> fetch Python files from that directory using HTTP requests, and finally >> load them as modules (or packages) in the running namespace. >> >> The repo containing a primitive (but working) version of the >> Finder/Loader, also contains self explanatory examples (in the README.md): >> >> https://github.com/operatorequals/httpimport >> >> >> My proposal is that this module can become a core Python feature, >> providing a way to load modules even from Github.com repositories, >> without the need to "git clone - setup.py install" them. >> >> >> Other languages, like golang, provide this functionality from their >> early days (day one?). Python development can be greatly improved if a >> "try before pip installing" mechanism gets in place, as it will add a >> lot to the REPL nature of the testing/experimenting process. > As a core feature? No no no no no no no no. Absolutely do NOT WANT > THIS. This is a security bug magnet; can you imagine trying to ensure > that malicious code is not executed, in an arbitrary execution > context? As an explicitly-enabled feature, it's a lot less hairy than > a permanently-active one (can you IMAGINE how terrifying that would > be?), but even so, trying to prove that addRemoteRepo (not a > PEP8-compliant name, btw) is getting the correct code is not going to > be easy. You have to (a) drop HTTP altogether and mandate SSL and (b) > be absolutely sure that your certificate chains are 100% dependable, > which - as we've seen recently - is a nontrivial task. > > The easiest way to add remote code is pip. For most packages, that's > what you want to be using: > > pip install requests > > will make "import requests" functional. I don't see pip mentioned > anywhere in your README, but you do mention the testing of pull > requests, so at very least, this wants some explanatory screed. > > But I'm not entirely sure I want to support this. You're explicitly > talking about using this with the creation of backdoors... in what, > exactly? What are you actually getting at here? > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ [View Less]

1 0

Remote package/module imports through HTTP/S
by John Torakis Aug. 23, 2017

Aug. 23, 2017

Hello all! Today I opened an issue in bugs.python.org (http://bugs.python.org/issue31264) proposing a module I created for remote package/module imports through standard HTTP/S. The concept is that, if a directory is served through HTTP/S (the way SimpleHTTPServer module serves directories), a Finder/Loader object can fetch Python files from that directory using HTTP requests, and finally load them as modules (or packages) in the running namespace. The repo containing a primitive (but … [View More]

3 2

Fwd: Please consider adding context manager versions of setUp/tearDown to unittest.TestCase
by Nick Coghlan Aug. 22, 2017

Aug. 22, 2017

Folks, this has come up before, but: please don't post through Google Groups, as it breaks everyone else's ability to easily reply to the entire mailing list. ---------- Forwarded message ---------- From: Nick Coghlan <ncoghlan(a)gmail.com> Date: 22 August 2017 at 15:32 Subject: Re: [Python-ideas] Please consider adding context manager versions of setUp/tearDown to unittest.TestCase To: Neil Girdhar <mistersheik(a)gmail.com> Cc: python-ideas <python-ideas(a)googlegroups.com> … [View More]

3 3

Re: [Python-ideas] How do you think about these language extensions?(Thautwarm)
by ?? ? Aug. 21, 2017

Aug. 21, 2017

Hi, all! I want to reply many people and it might annoy you if I write multiple replies... As a result, I write them all in one post. ---------------------------------------------------------------------------------- To Christopher Barker, Ph.D. ---------------------------------------------------------------------------------- Hi, Dr. Christopher Barker Ph.D. Just as what you said, > parentheses aren't that bad, and as far as I can tell, this is just another > way to call a function … [View More]on the results of a function. > The above is now spelled: > list(map(lambda x: x+2, range(5))) > which seems fine with me -- the only improvement I see is a more compact > way to spell lambda. (though really, a list comp is considered more > "pythonic" these days, yes? > [x+2 for x in range(5)] > nicely, we have list comps and generator expressions, so we can avoid the > list0 call. I'll try to talk something about why I think we need this grammar, the reasons for it are not just to remove parentheses. Could you please think this way to define a variable: >> var = expr() -> g1(_) if f(_) else g2(_) which equals >> test = f(expr()) >> var = g1(test) if f(test) else g2(test) which means that we have to use a temporary variable "test" to define "var". I think the second example is a bit lengthy, isn't it? The reason why I take this kind of grammar is that I can "flatten the programming logic". In another words,I can clearly state what I mean to say in order of my thinking. For example, >> lambda x: f(g(x)) -> map(_, range(100)) The codes above means that I'm stressing on what(an action) I'm going to do on an object "range(100)". However, sometimes the actions are not important, so if we want to stress on what we're going to do something on, we write this codes: >> range(100) -> map( lambda x:f(g(x)), _ ) Additionally, coding with chaining expressions makes me feel like writing a poem (Although it's a little difficult to me :) How do you think about writing the following codes? >> someone -> dosomething( _, options=options) \ -> is_meeting_some_conditions( _ ) \ -> result1() if _ else result2() where: options = ... result1 = lambda: ... result2 = lambda: ... def dosomething(obj, options) -> Any: ... def is_meeting_some_conditions( event : Any ) -> bool : ... In my opinion, it's quite readable and "smooth". To be honest, I think we can totolly do coding like chatting and it can be quite enjoyable. However, I'm not sure whether '->' is a good choice, and it didn't lead to any conflics at all when I compiled the CPython source code. Moreover, It can be easily changed in Grammar/Grammar, so I think it not to be crucial. Finally, > Also, we need to remember that functions can take *args, **kwargs, etc, > and can return a tuple of just about anything -- not sure how well that > maps to the "pipe" model. I think that using "pipe" model cannot be the right choice. We don't need to worry about this problem if we use the grammar I've implemented yet :) >> (lambda x: (x%5, x) ) -> max( range(99), key = _) >> 94 >> def max_from_seq(*args): return max(args) >> [1,2,3] -> max_from_seq(*_) >> 3 Thautwarm ---------------------------------------------------------------------------------- To David Mertz ---------------------------------------------------------------------------------- I think what you mean to is partially correct, but auto-currying cannot be effective enough. For sure you can do this in "compose" function. > ... -> ... -> ... -> map(lambda x:x+1, _) However, the evaluated result can be used as any argument of map and other callable objects. > ... -> ... -> ... -> map(_ , range(100)) > ... -> ... -> ... -> min([1,2,3], key = _ ) Thautwarm ---------------------------------------------------------------------------------- To Chris Angelico ---------------------------------------------------------------------------------- To be honest, I'm in favor of the grammar you prefer, like "expr1 | expr2 | expr3". However, it might be annoying that I should firstly define a lot of kinds of pipeline operators like Map, Reduce and so on. As for whether to allow implicit first/last argument, it seems to be a good idea but two points are in the way: 1. We need to change almost all C functions related to expressions in the source code located at Python/ast.c, while implementing the grammar I'm using now just needing nothing more than adding new C-function here. 2. Implicit methods makes it impossible to use following expressions. > ... -> func(some_var, some_var, _, some_eval(), some_key = _) In other words, implicit methods weaken the grammar, we need one more "where syntax" to do the same thing: > some -> new_func where: new_func = lambda x: func(some_var, some_var, x, some_eval(), some_key = x) Emmm... I'm not sure about that, how do you think about that? Thautwarm ---------------------------------------------------------------------------------- To Steven D'Aprano ---------------------------------------------------------------------------------- Thank you very much for your reply, and it encouraged me a lot. I've just read your most recent post, and it seems that you've suffered a lot from the parentheses, and so did I. > Half of my key presses are moving backwards over code I've just written > to insert a function call which is executed *after* what I wrote, but > needs to be written *before* what I just wrote. I couldn't agree more about what you've said here!!! My opinions about "chaining and pipeline" could be found in my reply to Chris Barker, sorry for that I could not repeat myself in the same post. >> # where syntax >> >> from math import pi >> r = 1 # the radius >> h = 10 # the height >> S = (2*S_top + S_side) where: >> S_top = pi*r**2 >> S_side = C * h where: >> C = 2*pi*r > This has been suggested a few times. The first time, I disliked it, but > I've come across to seeing its value. I like it. > I wonder: could we make the "where" clause delay evaluation until the > entire block was compiled, so that we could write something like this: > > S = (2*S_top + S_side) where: > S_top = pi*r**2 > S_side = C * h # C is defined further on > C = 2*pi*r > That's more how "where" is used mathematically. As far as I'm concerned, it's not sure to tell how about you opinion. The grammar you've just considered is quite Haskell-like, I think. And the reason why I want to use "where synatx" is to divide the programming logic clearly into different layers. For example, sometimes we just need to know that surface area of a cylinder is 2*S_top + S_side If someone see the codes, he may not need to know how S_top and S_side are evaluated,getting a knowledge of what it means to is enough. And if you want to get more about how to evaluate S_side and S_top, just see the next "where syntax" and find the answers. Here is another example, about forward propagation in neural networks. # input_layer[i] : "np.ndarray[:] " = np.array( ... ) # weight[i] : "np.ndarray[:][:]" = np.array( ... ) output_layer[i] = activate(input_layer[i]) where: """ logic layer 1 """ def activate( layer ): ... return activation[i](layer) # for example, activation[i] = lambda x:x input_layer[i] = forward(weight[i-1], output_layer[i-1].T) where: """ logic layer 2 """ def forward(weight, output): ... # if it's a normal multi-layer perceptron. return np.matmul(weight, output.T) For some people, their works just need them to know that forward propagation of a neural network means that the output layer is generated from the input layer with some transformation. For some people who want to know what the transformation is, they can go to the next "where syntax", and find the definition of the transformation which named "activate". For people who want to know how neural networks works with multiple layers, they can find that the input layer is defined by last output_layer, last weight matrix how NN forwards. I think it a good way to use "where syntax" to deconstruct the programming logic, which can strengthen the readability a lot! And then I'm going to talk something about Pattern Matching, and transform them to regular Python to make it clear to understand. >> Here is an example to use flowpython, which gives the permutations of a sequence. >> >> from copy import deepcopy >> permutations = .seq -> seq_seq where: >> condic+[] seq: >> case (a, ) => seq_seq = [a,] >> case (a, b) => seq_seq = [[a,b],[b,a]] >> case (a,*b) => >> seq_seq = permutations(b) -> map(.x -> insertAll(x, a), _) -> sum(_, []) where: >> insertAll = . x, a -> ret where: >> ret = [ deepcopy(x) -> _.insert(i, a) or _ for i in (len(x) -> range(_+1)) ] > I find that almost unreadable. Too many new features all at once, it's > like trying to read a completely unfamiliar language. > How would you translate that into regular Python? This algorithm can be fixed a little because the second case is redundant. And here is the regular Python codes transformed from the codes above. from copy import deepcopy def permutations(seq): try: # the first case (a, ) = seq return [a ,] except: try: # the third case (the second case is redundant) def insertAll(x, a): # insertAll([1,2,3], 0) -> [[0, 1, 2, 3], [1, 0, 2, 3], [1, 2, 0, 3], [1, 2, 3, 0]] ret = [] for i in range( len(x) + 1 ): tmp = deepcopy(x) tmp.insert(i, a) ret.append(tmp) return ret (a, *b) = seq tmp = permutations(b) tmp = map(lambda x : insertAll(x, a) , tmp) return sum(tmp, []) # sum([[1,2,3], [-1,-2,-3]], []) -> [1,2,3,-1,-2,-3] except: # no otherwise! pass To be continue...(sorry for my lack of time Thautwarm ---------------------------------------------------------------------------------- I'm sorry that I have to do some other works now and didn't finished writing down all I want to say. I'd like to continue replying the posts tomorrow , and it's quite glad to discuss these topics with you all!!! [View Less]

3 2

Please consider adding context manager versions of setUp/tearDown to unittest.TestCase
by Neil Girdhar Aug. 21, 2017

Aug. 21, 2017

This question describes an example of the problem: https://stackoverflow.com/questions/8416208/in-python-is-there-a-good-idiom…. You want to invoke a context manager in your setup/tearing-down, but the easiest way to do that is to override run, which seems ugly. Why not add two methods to unittest.TestCase whose default implementations are given below: class TestCase: @contextmanager def method_context(self): self.setUp() try: yield finally: … [View More]

3 2

How do you think about these language extensions?
by 王宣 ? Aug. 19, 2017

Aug. 19, 2017

Hi all, I've just finished a language extension for CPython 3.6.x to support some additional grammars like Pattern Matching. And It's compatible with CPython. I'm looking for constructive advice, and I wonder if you will be interested in this one. （ the project address is https://github.com/thautwarm/flowpython) [https://avatars1.githubusercontent.com/u/22536460?v=4&s=400]<https://github.com/thautwarm/flowpython> thautwarm/flowpython<https://github.com/thautwarm/flowpython> … [View More]github.com flowpython - tasty feature extensions for python(python3). Some examples here: # where syntax from math import pi r = 1 # the radius h = 10 # the height S = (2*S_top + S_side) where: S_top = pi*r**2 S_side = C * h where: C = 2*pi*r # lambda&curry : lambda x: lambda y: lambda z: ret where: ret = x+y ret -= z .x -> .y -> .z -> ret where: ret = x+y ret -= z as-with x def as y def as z def ret where: ret = x+y ret -= z # arrow transform (to avoid endless parentheses and try to be more readable. >> range(5) -> map(.x->x+2, _) -> list(_) >> [2,3,4,5,6] # pattern matching # use "condic" as keyword is for avoiding the conflictions against the standard libraries and packages from third party. "switch" and "match" both lead to conflictions. condic+(type) 1: case a:int => assert a == 1 and type(a) == 1 [>] case 0 => assert 1 > 0 [is not] case 1 => assert 1 is not 1 otherwise => print("nothing") condic+() [1,2,3]: case (a,*b)->b:list => sum(b) +[] case [] => print('empty list') +[==] case (a,b):(1,2) => print("the list is [1,2]") The grammars with more details and examples can be found in https://github.com/thautwarm/flowpython/wiki Does it interest you? If so, you can try it if you have CPython 3.6.x. pip install flowpython python -m flowpython -m enable/disable Here is an example to use flowpython, which gives the permutations of a sequence. from copy import deepcopy permutations = .seq -> seq_seq where: condic+[] seq: case (a, ) => seq_seq = [a,] case (a, b) => seq_seq = [[a,b],[b,a]] case (a,*b) => seq_seq = permutations(b) -> map(.x -> insertAll(x, a), _) -> sum(_, []) where: insertAll = . x, a -> ret where: ret = [ deepcopy(x) -> _.insert(i, a) or _ for i in (len(x) -> range(_+1)) ] If the object permutations are defined, try these codes in console: >> range(3) -> permutations(_) >> [[0, 1, 2], [1, 0, 2], [1, 2, 0], [0, 2, 1], [2, 0, 1], [2, 1, 0]] Does it seem to be interesting? Thanks, Thautwarm [View Less]

6 10

More Metadata for Variable Annotations
by Bagrat Aznauryan Aug. 18, 2017

Aug. 18, 2017

# Abstract Before the holly PEP-526 the only option for type hints were comments. And before PEP-484 the docstrings were the main place where variable metadata would go. That variable metadata would include: * the type * the human-readable description * some value constraints (e.g. a range for integer variable) PEP-526 introduced the awesome syntax sugar, which made the first part of the metadata - the type, easily introspectable during the runtime. However, if you still need to add the … [View More]description and the value constraints to the variable metadata, you still need to fallback to the docstring option. The idea is to make it possible to be able to include all of the mentioned metadata in the variable annotations. # Rationale Having the type specified using the supported annotation syntax and the rest of the metadata in the docstrings, adds duplication and complexity for further maintenance. Moreover, if you need the docstring-contained metadata to be used in the runtime, you need to implement a parser or pick one from existing ones which adds another dependency to your application. The need for the rest of the metadata other than the type, might be proven to be common. A typical example is generating the JSON Schema for a class, e.g. to be used for OpenAPI definition of your API. # Possible Solutions ## A wrapper The proposal is to introduce a new wrapper (probably a function), that will accept the type as the first positional argument and additional keyword arguments for metadata. The wrapper will map the keyword arguments to the type object as attributes and return it. The code would look like this: ``` foo: wrapper( int, description="bar", minimum=0, maximum=100 ) ``` Later, the metadata can be accessed as the annotation attributes, like e.g.: ``` __annotations__['foo'].description ``` ## Annotation as a tuple This solution does not require any code change in Python, but will force other tools change the parsing (e.g. mypy). The proposal is that when the annotation is optionally a tuple instance, use the first element as the type of the variable, and ignore the rest or treat as additional metadata. This will make it possible to add the metadata into a separate dictionary as the second element of the annotation tuple. For example: ``` foo: ( int, { description="bar", minimum=0, maximum=100 } ) ``` The annotation will be stored as is, so to access the metadata in the runtime, one would need to explicitly access the second item of the annotation tuple. # Summary This option would help to have a well annotated code which will be self-descriptive and provide abilities to generate schemas and other definitions (e.g. OpenAPI) automatically and without duplication. The proposed solutions are definitely not perfect and not the main point of this email. The target is to describe the idea and motivation and start a discussion. [View Less]

4 3

DOM syntax guide
by Abdur-Rahmaan Janhangeer Aug. 16, 2017

Aug. 16, 2017

greetings all, i like python and lot and would like to use it everywhere ... upto on the web (not django type). For python js-compiled versions (for makers) can you provide some syntax guidelines for dom access ? Abdur-Rahmaan Janhangeer, Mauritius abdurrahmaanjanhangeer.wordpress.com

3 5

New PEP 550: Execution Context
by Yury Selivanov Aug. 15, 2017

Aug. 15, 2017

Hi, This is a new PEP to implement Execution Contexts in Python. The PEP is in-flight to python.org, and in the meanwhile can be read on GitHub: https://github.com/python/peps/blob/master/pep-0550.rst (it contains a few diagrams and charts, so please read it there.) Thank you! Yury PEP: 550 Title: Execution Context Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov <yury(a)magic.io> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-Aug-2017 … [View More]Python-Version: 3.7 Post-History: 11-Aug-2017 Abstract ======== This PEP proposes a new mechanism to manage execution state--the logical environment in which a function, a thread, a generator, or a coroutine executes in. A few examples of where having a reliable state storage is required: * Context managers like decimal contexts, ``numpy.errstate``, and ``warnings.catch_warnings``; * Storing request-related data such as security tokens and request data in web applications; * Profiling, tracing, and logging in complex and large code bases. The usual solution for storing state is to use a Thread-local Storage (TLS), implemented in the standard library as ``threading.local()``. Unfortunately, TLS does not work for isolating state of generators or asynchronous code because such code shares a single thread. Rationale ========= Traditionally a Thread-local Storage (TLS) is used for storing the state. However, the major flaw of using the TLS is that it works only for multi-threaded code. It is not possible to reliably contain the state within a generator or a coroutine. For example, consider the following generator:: def calculate(precision, ...): with decimal.localcontext() as ctx: # Set the precision for decimal calculations # inside this block ctx.prec = precision yield calculate_something() yield calculate_something_else() Decimal context is using a TLS to store the state, and because TLS is not aware of generators, the state can leak. The above code will not work correctly, if a user iterates over the ``calculate()`` generator with different precisions in parallel:: g1 = calculate(100) g2 = calculate(50) items = list(zip(g1, g2)) # items[0] will be a tuple of: # first value from g1 calculated with 100 precision, # first value from g2 calculated with 50 precision. # # items[1] will be a tuple of: # second value from g1 calculated with 50 precision, # second value from g2 calculated with 50 precision. An even scarier example would be using decimals to represent money in an async/await application: decimal calculations can suddenly lose precision in the middle of processing a request. Currently, bugs like this are extremely hard to find and fix. Another common need for web applications is to have access to the current request object, or security context, or, simply, the request URL for logging or submitting performance tracing data:: async def handle_http_request(request): context.current_http_request = request await ... # Invoke your framework code, render templates, # make DB queries, etc, and use the global # 'current_http_request' in that code. # This isn't currently possible to do reliably # in asyncio out of the box. These examples are just a few out of many, where a reliable way to store context data is absolutely needed. The inability to use TLS for asynchronous code has lead to proliferation of ad-hoc solutions, limited to be supported only by code that was explicitly enabled to work with them. Current status quo is that any library, including the standard library, that uses a TLS, will likely not work as expected in asynchronous code or with generators (see [3]_ as an example issue.) Some languages that have coroutines or generators recommend to manually pass a ``context`` object to every function, see [1]_ describing the pattern for Go. This approach, however, has limited use for Python, where we have a huge ecosystem that was built to work with a TLS-like context. Moreover, passing the context explicitly does not work at all for libraries like ``decimal`` or ``numpy``, which use operator overloading. .NET runtime, which has support for async/await, has a generic solution of this problem, called ``ExecutionContext`` (see [2]_). On the surface, working with it is very similar to working with a TLS, but the former explicitly supports asynchronous code. Goals ===== The goal of this PEP is to provide a more reliable alternative to ``threading.local()``. It should be explicitly designed to work with Python execution model, equally supporting threads, generators, and coroutines. An acceptable solution for Python should meet the following requirements: * Transparent support for code executing in threads, coroutines, and generators with an easy to use API. * Negligible impact on the performance of the existing code or the code that will be using the new mechanism. * Fast C API for packages like ``decimal`` and ``numpy``. Explicit is still better than implicit, hence the new APIs should only be used when there is no option to pass the state explicitly. With this PEP implemented, it should be possible to update a context manager like the below:: _local = threading.local() @contextmanager def context(x): old_x = getattr(_local, 'x', None) _local.x = x try: yield finally: _local.x = old_x to a more robust version that can be reliably used in generators and async/await code, with a simple transformation:: @contextmanager def context(x): old_x = get_execution_context_item('x') set_execution_context_item('x', x) try: yield finally: set_execution_context_item('x', old_x) Specification ============= This proposal introduces a new concept called Execution Context (EC), along with a set of Python APIs and C APIs to interact with it. EC is implemented using an immutable mapping. Every modification of the mapping produces a new copy of it. To illustrate what it means let's compare it to how we work with tuples in Python:: a0 = () a1 = a0 + (1,) a2 = a1 + (2,) # a0 is an empty tuple # a1 is (1,) # a2 is (1, 2) Manipulating an EC object would be similar:: a0 = EC() a1 = a0.set('foo', 'bar') a2 = a1.set('spam', 'ham') # a0 is an empty mapping # a1 is {'foo': 'bar'} # a2 is {'foo': 'bar', 'spam': 'ham'} In CPython, every thread that can execute Python code has a corresponding ``PyThreadState`` object. It encapsulates important runtime information like a pointer to the current frame, and is being used by the ceval loop extensively. We add a new field to ``PyThreadState``, called ``exec_context``, which points to the current EC object. We also introduce a set of APIs to work with Execution Context. In this section we will only cover two functions that are needed to explain how Execution Context works. See the full list of new APIs in the `New APIs`_ section. * ``sys.get_execution_context_item(key, default=None)``: lookup ``key`` in the EC of the executing thread. If not found, return ``default``. * ``sys.set_execution_context_item(key, value)``: get the current EC of the executing thread. Add a ``key``/``value`` item to it, which will produce a new EC object. Set the new object as the current one for the executing thread. In pseudo-code:: tstate = PyThreadState_GET() ec = tstate.exec_context ec2 = ec.set(key, value) tstate.exec_context = ec2 Note, that some important implementation details and optimizations are omitted here, and will be covered in later sections of this PEP. Now let's see how Execution Contexts work with regular multi-threaded code, generators, and coroutines. Regular & Multithreaded Code ---------------------------- For regular Python code, EC behaves just like a thread-local. Any modification of the EC object produces a new one, which is immediately set as the current one for the thread state. .. figure:: pep-0550/functions.png :align: center :width: 90% Figure 1. Execution Context flow in a thread. As Figure 1 illustrates, if a function calls ``set_execution_context_item()``, the modification of the execution context will be visible to all subsequent calls and to the caller:: def set_foo(): set_execution_context_item('foo', 'spam') set_execution_context_item('foo', 'bar') print(get_execution_context_item('foo')) set_foo() print(get_execution_context_item('foo')) # will print: # bar # spam Coroutines ---------- Python :pep:`492` coroutines are used to implement cooperative multitasking. For a Python end-user they are similar to threads, especially when it comes to sharing resources or modifying the global state. An event loop is needed to schedule coroutines. Coroutines that are explicitly scheduled by the user are usually called Tasks. When a coroutine is scheduled, it can schedule other coroutines using an ``await`` expression. In async/await world, awaiting a coroutine can be viewed as a different calling convention: Tasks are similar to threads, and awaiting on coroutines within a Task is similar to calling functions within a thread. By drawing a parallel between regular multithreaded code and async/await, it becomes apparent that any modification of the execution context within one Task should be visible to all coroutines scheduled within it. Any execution context modifications, however, must not be visible to other Tasks executing within the same thread. To achieve this, a small set of modifications to the coroutine object is needed: * When a coroutine object is instantiated, it saves a reference to the current execution context object to its ``cr_execution_context`` attribute. * Coroutine's ``.send()`` and ``.throw()`` methods are modified as follows (in pseudo-C):: if coro->cr_isolated_execution_context: # Save a reference to the current execution context old_context = tstate->execution_context # Set our saved execution context as the current # for the current thread. tstate->execution_context = coro->cr_execution_context try: # Perform the actual `Coroutine.send()` or # `Coroutine.throw()` call. return coro->send(...) finally: # Save a reference to the updated execution_context. # We will need it later, when `.send()` or `.throw()` # are called again. coro->cr_execution_context = tstate->execution_context # Restore thread's execution context to what it was before # invoking this coroutine. tstate->execution_context = old_context else: # Perform the actual `Coroutine.send()` or # `Coroutine.throw()` call. return coro->send(...) * ``cr_isolated_execution_context`` is a new attribute on coroutine objects. Set to ``True`` by default, it makes any execution context modifications performed by coroutine to stay visible only to that coroutine. When Python interpreter sees an ``await`` instruction, it flips ``cr_isolated_execution_context`` to ``False`` for the coroutine that is about to be awaited. This makes any changes to execution context made by nested coroutine calls within a Task to be visible throughout the Task. Because the top-level coroutine (Task) cannot be scheduled with ``await`` (in asyncio you need to call ``loop.create_task()`` or ``asyncio.ensure_future()`` to schedule a Task), all execution context modifications are guaranteed to stay within the Task. * We always work with ``tstate->exec_context``. We use ``coro->cr_execution_context`` only to store coroutine's execution context when it is not executing. Figure 2 below illustrates how execution context mutations work with coroutines. .. figure:: pep-0550/coroutines.png :align: center :width: 90% Figure 2. Execution Context flow in coroutines. In the above diagram: * When "coro1" is created, it saves a reference to the current execution context "2". * If it makes any change to the context, it will have its own execution context branch "2.1". * When it awaits on "coro2", any subsequent changes it does to the execution context are visible to "coro1", but not outside of it. In code:: async def inner_foo(): print('inner_foo:', get_execution_context_item('key')) set_execution_context_item('key', 2) async def foo(): print('foo:', get_execution_context_item('key')) set_execution_context_item('key', 1) await inner_foo() print('foo:', get_execution_context_item('key')) set_execution_context_item('key', 'spam') print('main:', get_execution_context_item('key')) asyncio.get_event_loop().run_until_complete(foo()) print('main:', get_execution_context_item('key')) which will output:: main: spam foo: spam inner_foo: 1 foo: 2 main: spam Generator-based coroutines (generators decorated with ``types.coroutine`` or ``asyncio.coroutine``) behave exactly as native coroutines with regards to execution context management: their ``yield from`` expression is semantically equivalent to ``await``. Generators ---------- Generators in Python, while similar to Coroutines, are used in a fundamentally different way. They are producers of data, and they use ``yield`` expression to suspend/resume their execution. A crucial difference between ``await coro`` and ``yield value`` is that the former expression guarantees that the ``coro`` will be executed to the end, while the latter is producing ``value`` and suspending the generator until it gets iterated again. Generators share 99% of their implementation with coroutines, and thus have similar new attributes ``gi_execution_context`` and ``gi_isolated_execution_context``. Similar to coroutines, generators save a reference to the current execution context when they are instantiated. The have the same implementation of ``.send()`` and ``.throw()`` methods. The only difference is that ``gi_isolated_execution_context`` is always set to ``True``, and is never modified by the interpreter. ``yield from o`` expression in regular generators that are not decorated with ``types.coroutine``, is semantically equivalent to ``for v in o: yield v``. .. figure:: pep-0550/generators.png :align: center :width: 90% Figure 3. Execution Context flow in a generator. In the above diagram: * When "gen1" is created, it saves a reference to the current execution context "2". * If it makes any change to the context, it will have its own execution context branch "2.1". * When "gen2" is created, it saves a reference to the current execution context for it -- "2.1". * Any subsequent execution context updated in "gen2" will only be visible to "gen2". * Likewise, any context changes that "gen1" will do after it created "gen2" will not be visible to "gen2". In code:: def inner_foo(): for i in range(3): print('inner_foo:', get_execution_context_item('key')) set_execution_context_item('key', i) yield i def foo(): set_execution_context_item('key', 'spam') print('foo:', get_execution_context_item('key')) inner = inner_foo() while True: val = next(inner, None) if val is None: break yield val print('foo:', get_execution_context_item('key')) set_execution_context_item('key', 'spam') print('main:', get_execution_context_item('key')) list(foo()) print('main:', get_execution_context_item('key')) which will output:: main: ham foo: spam inner_foo: spam foo: spam inner_foo: 0 foo: spam inner_foo: 1 foo: spam main: ham As we see, any modification of the execution context in a generator is visible only to the generator itself. There is one use-case where it is desired for generators to affect the surrounding execution context: ``contextlib.contextmanager`` decorator. To make the following work:: @contextmanager def context(x): old_x = get_execution_context_item('x') set_execution_context_item('x', x) try: yield finally: set_execution_context_item('x', old_x) we modified ``contextmanager`` to flip ``gi_isolated_execution_context`` flag to ``False`` on its generator. Greenlets --------- Greenlet is an alternative implementation of cooperative scheduling for Python. Although greenlet package is not part of CPython, popular frameworks like gevent rely on it, and it is important that greenlet can be modified to support execution contexts. In a nutshell, greenlet design is very similar to design of generators. The main difference is that for generators, the stack is managed by the Python interpreter. Greenlet works outside of the Python interpreter, and manually saves some ``PyThreadState`` fields and pushes/pops the C-stack. Since Execution Context is implemented on top of ``PyThreadState``, it's easy to add transparent support of it to greenlet. New APIs ======== Even though this PEP adds a number of new APIs, please keep in mind, that most Python users will likely ever use only two of them: ``sys.get_execution_context_item()`` and ``sys.set_execution_context_item()``. Python ------ 1. ``sys.get_execution_context_item(key, default=None)``: lookup ``key`` for the current Execution Context. If not found, return ``default``. 2. ``sys.set_execution_context_item(key, value)``: set ``key``/``value`` item for the current Execution Context. If ``value`` is ``None``, the item will be removed. 3. ``sys.get_execution_context()``: return the current Execution Context object: ``sys.ExecutionContext``. 4. ``sys.set_execution_context(ec)``: set the passed ``sys.ExecutionContext`` instance as a current one for the current thread. 5. ``sys.ExecutionContext`` object. Implementation detail: ``sys.ExecutionContext`` wraps a low-level ``PyExecContextData`` object. ``sys.ExecutionContext`` has a mutable mapping API, abstracting away the real immutable ``PyExecContextData``. * ``ExecutionContext()``: construct a new, empty, execution context. * ``ec.run(func, *args)`` method: run ``func(*args)`` in the ``ec`` execution context. * ``ec[key]``: lookup ``key`` in ``ec`` context. * ``ec[key] = value``: assign ``key``/``value`` item to the ``ec``. * ``ec.get()``, ``ec.items()``, ``ec.values()``, ``ec.keys()``, and ``ec.copy()`` are similar to that of ``dict`` object. C API ----- C API is different from the Python one because it operates directly on the low-level immutable ``PyExecContextData`` object. 1. New ``PyThreadState->exec_context`` field, pointing to a ``PyExecContextData`` object. 2. ``PyThreadState_SetExecContextItem`` and ``PyThreadState_GetExecContextItem`` similar to ``sys.set_execution_context_item()`` and ``sys.get_execution_context_item()``. 3. ``PyThreadState_GetExecContext``: similar to ``sys.get_execution_context()``. Always returns an ``PyExecContextData`` object. If ``PyThreadState->exec_context`` is ``NULL`` an new and empty one will be created and assigned to ``PyThreadState->exec_context``. 4. ``PyThreadState_SetExecContext``: similar to ``sys.set_execution_context()``. 5. ``PyExecContext_New``: create a new empty ``PyExecContextData`` object. 6. ``PyExecContext_SetItem`` and ``PyExecContext_GetItem``. The exact layout ``PyExecContextData`` is private, which allows to switch it to a different implementation later. More on that in the `Implementation Details`_ section. Modifications in Standard Library ================================= * ``contextlib.contextmanager`` was updated to flip the new ``gi_isolated_execution_context`` attribute on the generator. * ``asyncio.events.Handle`` object now captures the current execution context when it is created, and uses the saved execution context to run the callback (with ``ExecutionContext.run()`` method.) This makes ``loop.call_soon()`` to run callbacks in the execution context they were scheduled. No modifications in ``asyncio.Task`` or ``asyncio.Future`` were necessary. Some standard library modules like ``warnings`` and ``decimal`` can be updated to use new execution contexts. This will be considered in separate issues if this PEP is accepted. Backwards Compatibility ======================= This proposal preserves 100% backwards compatibility. Performance =========== Implementation Details ---------------------- The new ``PyExecContextData`` object is wrapping a ``dict`` object. Any modification requires creating a shallow copy of the dict. While working on the reference implementation of this PEP, we were able to optimize ``dict.copy()`` operation **5.5x**, see [4]_ for details. .. figure:: pep-0550/dict_copy.png :align: center :width: 100% Figure 4. Figure 4 shows that the performance of immutable dict implemented with shallow copying is expectedly O(n) for the ``set()`` operation. However, this is tolerable until dict has more than 100 items (1 ``set()`` takes about a microsecond.) Judging by the number of modules that need EC in Standard Library it is likely that real world Python applications will use significantly less than 100 execution context variables. The important point is that the cost of accessing a key in Execution Context is always O(1). If the ``set()`` operation performance is a major concern, we discuss alternative approaches that have O(1) or close ``set()`` performance in `Alternative Immutable Dict Implementation`_, `Faster C API`_, and `Copy-on-write Execution Context`_ sections. Generators and Coroutines ------------------------- Using a microbenchmark for generators and coroutines from :pep:`492` ([12]_), it was possible to observe 0.5 to 1% performance degradation. asyncio echoserver microbechmarks from the uvloop project [13]_ showed 1-1.5% performance degradation for asyncio code. asyncpg benchmarks [14]_, that execute more code and are closer to a real-world application did not exhibit any noticeable performance change. Overall Performance Impact -------------------------- The total number of changed lines in the ceval loop is 2 -- in the ``YIELD_FROM`` opcode implementation. Only performance of generators and coroutines can be affected by the proposal. This was confirmed by running Python Performance Benchmark Suite [15]_, which demonstrated that there is no difference between 3.7 master branch and this PEP reference implementation branch (full benchmark results can be found here [16]_.) Design Considerations ===================== Alternative Immutable Dict Implementation ----------------------------------------- Languages like Clojure and Scala use Hash Array Mapped Tries (HAMT) to implement high performance immutable collections [5]_, [6]_. Immutable mappings implemented with HAMT have O(log\ :sub:`32`\ N) performance for both ``set()`` and ``get()`` operations, which will be essentially O(1) for relatively small mappings in EC. To assess if HAMT can be used for Execution Context, we implemented it in CPython [7]_. .. figure:: pep-0550/hamt_vs_dict.png :align: center :width: 100% Figure 5. Benchmark code can be found here: [9]_. Figure 5 shows that HAMT indeed displays O(1) performance for all benchmarked dictionary sizes. For dictionaries with less than 100 items, HAMT is a bit slower than Python dict/shallow copy. .. figure:: pep-0550/lookup_hamt.png :align: center :width: 100% Figure 6. Benchmark code can be found here: [10]_. Figure 6 below shows comparison of lookup costs between Python dict and an HAMT immutable mapping. HAMT lookup time is 30-40% worse than Python dict lookups on average, which is a very good result, considering how well Python dicts are optimized. Note, that according to [8]_, HAMT design can be further improved. The bottom line is that the current approach with implementing an immutable mapping with shallow-copying dict will likely perform adequately in real-life applications. The HAMT solution is more future proof, however. The proposed API is designed in such a way that the underlying implementation of the mapping can be changed completely without affecting the Execution Context `Specification`_, which allows us to switch to HAMT at some point if necessary. Copy-on-write Execution Context ------------------------------- The implementation of Execution Context in .NET is different from this PEP. .NET uses copy-on-write mechanism and a regular mutable mapping. One way to implement this in CPython would be to have two new fields in ``PyThreadState``: * ``exec_context`` pointing to the current Execution Context mapping; * ``exec_context_copy_on_write`` flag, set to ``0`` initially. The idea is that whenever we are modifying the EC, the copy-on-write flag is checked, and if it is set to ``1``, the EC is copied. Modifications to Coroutine and Generator ``.send()`` and ``.throw()`` methods described in the `Coroutines`_ section will be almost the same, except that in addition to the ``gi_execution_context`` they will have a ``gi_exec_context_copy_on_write`` flag. When a coroutine or a generator starts, the flag will be set to ``1``. This will ensure that any modification of the EC performed within a coroutine or a generator will be isolated. This approach has one advantage: * For Execution Context that contains a large number of items, copy-on-write is a more efficient solution than the shallow-copy dict approach. However, we believe that copy-on-write disadvantages are more important to consider: * Copy-on-write behaviour for generators and coroutines makes EC semantics less predictable. With immutable EC approach, generators and coroutines always execute in the EC that was current at the moment of their creation. Any modifications to the outer EC while a generator or a coroutine is executing are not visible to them:: def generator(): yield 1 print(get_execution_context_item('key')) yield 2 set_execution_context_item('key', 'spam') gen = iter(generator()) next(gen) set_execution_context_item('key', 'ham') next(gen) The above script will always print 'spam' with immutable EC. With a copy-on-write approach, the above script will print 'ham'. Now, consider that ``generator()`` was refactored to call some library function, that uses Execution Context:: def generator(): yield 1 some_function_that_uses_decimal_context() print(get_execution_context_item('key')) yield 2 Now, the script will print 'spam', because ``some_function_that_uses_decimal_context`` forced the EC to copy, and ``set_execution_context_item('key', 'ham')`` line did not affect the ``generator()`` code after all. * Similarly to the previous point, ``sys.ExecutionContext.run()`` method will also become less predictable, as ``sys.get_execution_context()`` would still return a reference to the current mutable EC. We can't modify ``sys.get_execution_context()`` to return a shallow copy of the current EC, because this would seriously harm performance of ``asyncio.call_soon()`` and similar places, where it is important to propagate the Execution Context. * Even though copy-on-write requires to shallow copy the execution context object less frequently, copying will still take place in coroutines and generators. In which case, HAMT approach will perform better for medium to large sized execution contexts. All in all, we believe that the copy-on-write approach introduces very subtle corner cases that could lead to bugs that are exceptionally hard to discover and fix. The immutable EC solution in comparison is always predictable and easy to reason about. Therefore we believe that any slight performance gain that the copy-on-write solution might offer is not worth it. Faster C API ------------ Packages like numpy and standard library modules like decimal need to frequently query the global state for some local context configuration. It is important that the APIs that they use is as fast as possible. The proposed ``PyThreadState_SetExecContextItem`` and ``PyThreadState_GetExecContextItem`` functions need to get the current thread state with ``PyThreadState_GET()`` (fast) and then perform a hash lookup (relatively slow). We can eliminate the hash lookup by adding three additional C API functions: * ``Py_ssize_t PyExecContext_RequestIndex(char *key_name)``: a function similar to the existing ``_PyEval_RequestCodeExtraIndex`` introduced :pep:`523`. The idea is to request a unique index that can later be used to lookup context items. The ``key_name`` can later be used by ``sys.ExecutionContext`` to introspect items added with this API. * ``PyThreadState_SetExecContextIndexedItem(Py_ssize_t index, PyObject *val)`` and ``PyThreadState_GetExecContextIndexedItem(Py_ssize_t index)`` to request an item by its index, avoiding the cost of hash lookup. Why setting a key to None removes the item? ------------------------------------------- Consider a context manager:: @contextmanager def context(x): old_x = get_execution_context_item('x') set_execution_context_item('x', x) try: yield finally: set_execution_context_item('x', old_x) With ``set_execution_context_item(key, None)`` call removing the ``key``, the user doesn't need to write additional code to remove the ``key`` if it wasn't in the execution context already. An alternative design with ``del_execution_context_item()`` method would look like the following:: @contextmanager def context(x): not_there = object() old_x = get_execution_context_item('x', not_there) set_execution_context_item('x', x) try: yield finally: if old_x is not_there: del_execution_context_item('x') else: set_execution_context_item('x', old_x) Can we fix ``PyThreadState_GetDict()``? --------------------------------------- ``PyThreadState_GetDict`` is a TLS, and some of its existing users might depend on it being just a TLS. Changing its behaviour to follow the Execution Context semantics would break backwards compatibility. PEP 521 ------- :pep:`521` proposes an alternative solution to the problem: enhance Context Manager Protocol with two new methods: ``__suspend__`` and ``__resume__``. To make it compatible with async/await, the Asynchronous Context Manager Protocol will also need to be extended with ``__asuspend__`` and ``__aresume__``. This allows to implement context managers like decimal context and ``numpy.errstate`` for generators and coroutines. The following code:: class Context: def __enter__(self): self.old_x = get_execution_context_item('x') set_execution_context_item('x', 'something') def __exit__(self, *err): set_execution_context_item('x', self.old_x) would become this:: class Context: def __enter__(self): self.old_x = get_execution_context_item('x') set_execution_context_item('x', 'something') def __suspend__(self): set_execution_context_item('x', self.old_x) def __resume__(self): set_execution_context_item('x', 'something') def __exit__(self, *err): set_execution_context_item('x', self.old_x) Besides complicating the protocol, the implementation will likely negatively impact performance of coroutines, generators, and any code that uses context managers, and will notably complicate the interpreter implementation. It also does not solve the leaking state problem for greenlet/gevent. :pep:`521` also does not provide any mechanism to propagate state in a local context, like storing a request object in an HTTP request handler to have better logging. Can Execution Context be implemented outside of CPython? -------------------------------------------------------- Because async/await code needs an event loop to run it, an EC-like solution can be implemented in a limited way for coroutines. Generators, on the other hand, do not have an event loop or trampoline, making it impossible to intercept their ``yield`` points outside of the Python interpreter. Reference Implementation ======================== The reference implementation can be found here: [11]_. References ========== .. [1] https://blog.golang.org/context .. [2] https://msdn.microsoft.com/en-us/library/system.threading.executioncontext.… .. [3] https://github.com/numpy/numpy/issues/9444 .. [4] http://bugs.python.org/issue31179 .. [5] https://en.wikipedia.org/wiki/Hash_array_mapped_trie .. [6] http://blog.higher-order.net/2010/08/16/assoc-and-clojures-persistenthashma… .. [7] https://github.com/1st1/cpython/tree/hamt .. [8] https://michael.steindorfer.name/publications/oopsla15.pdf .. [9] https://gist.github.com/1st1/9004813d5576c96529527d44c5457dcd .. [10] https://gist.github.com/1st1/dbe27f2e14c30cce6f0b5fddfc8c437e .. [11] https://github.com/1st1/cpython/tree/pep550 .. [12] https://www.python.org/dev/peps/pep-0492/#async-await .. [13] https://github.com/MagicStack/uvloop/blob/master/examples/bench/echoserver.… .. [14] https://github.com/MagicStack/pgbench .. [15] https://github.com/python/performance .. [16] https://gist.github.com/1st1/6b7a614643f91ead3edf37c4451a6b4c Copyright ========= This document has been placed in the public domain. [View Less]

11 44

Re: [Python-ideas] New PEP 550: Execution Context
by Yury Selivanov Aug. 13, 2017

Aug. 13, 2017

[replying to the list] On Sun, Aug 13, 2017 at 6:14 AM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > On 13 August 2017 at 16:01, Yury Selivanov <yselivanov.ml(a)gmail.com> wrote: >> On Sat, Aug 12, 2017 at 10:56 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: >> [..] >>> As Nathaniel suggestion, getting/setting/deleting individual items in >>> the current context would be implemented as methods on the ContextItem >>> objects, allowing … [View More]the return value of "get_context_items" to be a >>> plain dictionary, rather than a special type that directly supported >>> updates to the underlying context. >> >> The current PEP 550 design returns a "snapshot" of the current EC with >> sys.get_execution_context(). >> >> I.e. if you do >> >> ec = sys.get_execution_context() >> ec['a'] = 'b' >> >> # sys.get_execution_context_item('a') will return None >> >> You did get a snapshot and you modified it -- but your modifications >> are not visible anywhere. You can run a function in that modified EC >> with `ec.run(function)` and that function will see that new 'a' key, >> but that's it. There's no "magical" updates to the underlying context. > > In that case, I think "get_execution_context()" is quite misleading as > a name, and is going to be prone to exactly the confusion we currently > have with the mapping returned by locals(), which is that regardless > of whether writes to it affect the target namespace or not, it's going > to be surprising in at least some situations. > > So despite being initially in favour of exposing a mapping-like API at > the Python level, I'm now coming around to Armin Ronacher's point of > view: the copy-on-write semantics for the active context are > sufficiently different from any other mapping type in Python that we > should just avoid the use of __setitem__ and __delitem__ as syntactic > sugar entirely. I agree. I'll be redesigning the PEP to use the following API (please ignore the naming peculiarities, there are so many proposals at this point that I'll just stick to something I have in my head): 1. sys.new_execution_context_key('description') -> sys.ContextItem (or maybe we should just expose the sys.ContextItem type and let people instantiate it?) A key (or "token") to use with the execution context. Besides eliminating the names collision issue, it'll also have a slightly better performance, because its __hash__ method will always return a constant. (Strings cache their __hash__, but other types don't). 2. ContextItem.has(), ContextItem.get(), ContextItem.set(), ContextItem.delete() -- pretty self-explanatory. 3. sys.get_active_context() -> sys.ExecutionContext -- an immutable object, has no methods to modify the context. 3a. sys.ExecutionContext.run(callable, *args) -- run a callable(*args) in some execution context. 3b. sys.ExecutionContext.items() -- an iterator of ContextItem -> value for introspection and debugging purposes. 4. No sys.set_execution_context() method. At this point I'm not sure it's a good idea to allow users to change the current execution context to something else entirely. For use cases like enabling concurrent.futures to run your function within the current EC, you just use the sys.get_active_context()/ExecutionContext.run combination. If anything, we can add this function later. > Instead, we'd lay out the essential primitive operations that *only* > the interpreter can provide and define procedural interfaces for > those, and if anyone wanted to build a higher level object-oriented > interface on top of those primitives, they'd be free to do so, with > the procedural API acting as the abstraction layer that decouples "how > interpreters actually implement it" (e.g. copy-on-write mappings) from > "how libraries and frameworks model it for their own use" (e.g. rich > application context objects). That way, each interpreter would also be > free to define their *internal* object model in whichever way made the > most sense for them, rather than enshrining a point-in-time snaphot of > CPython's preferred implementation model as part of the language > definition. I agree. I like that this idea gives us more flexibility with the exact implementation strategy. [..] > The essential capabilities for active context manipulation would then be: > > - get_active_context_token() > - set_active_context(context_token) As I mentioned above, at this point I'm not entirely sure that we even need "set_active_context". The only useful thing for it that I can imagine is creating a decorator that isolates any changes of the context, but the only usecase for this I see is unittests. But even for unittests, a better solution is to use a decorator that detects keys that were added but not deleted during the test (leaks). > - implicitly saving and reverting the active context around various operations Usually we need to save/revert one particular context item, not the whole context. > - accessing the active context id for suspended coroutines and > generators (so parent contexts can opt-in to seeing changes made in > child contexts) Yes, this might be useful, let's keep it. > > Running commands in a particular context *wouldn't* be a primitive > operation given those building blocks, since you can implement that > for yourself using the above primitives: > > def run_in_context(target_context_token, func, *args, **kwds): > old_context_token = get_active_context_token() > set_active_context(target_context_token) > try: > func(*args, **kwds) > finally: > set_active_context(old_context_token) I'd still prefer to implement this as part of the spec. There are some tricks that I want to use to make ExecutionContext.run() much faster than a pure Python version. This is a highly performance critical part of the PEP -- call_soon in asyncio is a VERY frequent thing. Besides, having ExecutionContext.run eliminates the need to sys.set_active_context() -- again, we need to discuss this, but I see less and less utility for it now. > > The public manipulation API here would be deliberately based on opaque > tokens to make it clear that creating and mutating execution contexts > is entirely within the realm of the interpreter implementation, and > user level code can only control *which* execution context is active > in the current thread, not create arbitrary new execution contexts of > its own (at least, not without writing a CPython-specific C > extension). > > For manipulation of values within the active context, looking at other > comparable APIs, I think the main prior art within the language would > be: > > 1. threading.local(), which uses the descriptor protocol to handle > arbitrary attributes > 2. Cell variable references in function `__closure__` attributes, > which also uses the descriptor protocol by way of the "cell_contents" > attribute > > In 3.7, those two examples are being brought closer by way of > `cell_contents` becoming a read/write attribute: > > >>> def f(i): > ... def g(): > ... nonlocal i > ... return i > ... return g > ... > >>> g = f(0) > >>> g() > 0 > >>> cell = g.__closure__[0] > >>> cell.cell_contents > 0 > >>> cell.cell_contents = 5 > >>> g() > 5 > >>> del cell.cell_contents > >>> g() > Traceback (most recent call last): > ... > NameError: free variable 'i' referenced before assignment in enclosing scope > >>> cell.cell_contents = 0 > >>> g() > 0 > > This is very similar to the way manipulation of entries within a > thread local namespace works, but with each cell containing exactly > one attribute. > > For context items, I agree with Nathaniel that the cell-style > one-value-per-item approach is likely to be the way to go. To > emphasise that changes to that attribute only affect the *active* > context, I think "active_value" would be a good name: > > >>> request_id = > sys.create_context_item("my_web_framework.request_id", "Request > identifier for my_web_framework") > >>> request_id.active_value > Traceback (most recent call last): > ... > RuntimeError: Context item "my_web_framework.request" not set in > context <context token> > >>> request_id.active_value = "12345" > >>> request_id.active_value > '12345' I myself prefer a functional API to to __getattr__. I don't like the "del local.x" syntax. I don't think we are forced to follow the threading.local() API here, aren't we? Yury > > Finally, given opaque context tokens, and context items that worked > like closure cells (only accessing the active context rather than > lexically scoped variables), the one introspection primitive the > *interpreter* would need to provide is either: > > 1. Given a context token, return a mapping from context items to their > defined values in the given context > 2. A way to get a listing of the context items defined in the active context > > Since either of those can be defined in terms of the other, my own > preference goes to the first one, since using it to implement the > second alternative just requires a simple > `sys.get_active_context_token()` call, while implementing the first > one in terms of the second one requires a helper like > `run_in_context()` above to manipulate the active context in the > current thread. > > The first one also makes it fairly straightforward to *diff* a given > context against the active one - get the mappings for both contexts, > check which keys they have in common, compare the values for the > common keys, and then report on > > - keys that appear in one context but not the other > - values which differ between them for common keys > - (optionally) values which are the same for common keys > > Cheers, > Nick. [View Less]

2 2

Re: [Python-ideas] Python-ideas Digest, Vol 129, Issue 44
by 王宣赵 Aug. 13, 2017

Aug. 13, 2017

Thank you for your consideration. 获取 Outlook for Android<https://aka.ms/ghei36> 发件人: python-ideas-request(a)python.org 发送时间: 8月14日星期一 03:14 主题: Python-ideas Digest, Vol 129, Issue 44 收件人: python-ideas(a)python.org Send Python-ideas mailing list submissions to python-ideas(a)python.org To subscribe or unsubscribe via the World Wide Web, visit https://mail.python.org/mailman/listinfo/python-ideas or, via email, send a message with subject or body 'help' to python-ideas-request(a)… [View More]python.org You can reach the person managing the list at python-ideas-owner(a)python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Python-ideas digest..." Today's Topics: 1. How do you think about these language extensions? (?? ?) 2. Re: New PEP 550: Execution Context (Yury Selivanov) 3. Re: New PEP 550: Execution Context (Nathaniel Smith) ---------------------------------------------------------------------- Message: 1 Date: Sun, 13 Aug 2017 12:49:45 +0000 From: ?? ? To: "python-ideas(a)python.org" Subject: [Python-ideas] How do you think about these language extensions? Message-ID: Content-Type: text/plain; charset="iso-2022-jp" Hi all, I've just finished a language extension for CPython 3.6.x to support some additional grammars like Pattern Matching. And It's compatible with CPython. I'm looking for constructive advice, and I wonder if you will be interested in this one. ? the project address is https://github.com/thautwarm/flowpython) [https://avatars1.githubusercontent.com/u/22536460?v=4&s=400] thautwarm/flowpython github.com flowpython - tasty feature extensions for python(python3). Some examples here: # where syntax from math import pi r = 1 # the radius h = 10 # the height S = (2*S_top + S_side) where: S_top = pi*r**2 S_side = C * h where: C = 2*pi*r # lambda&curry : lambda x: lambda y: lambda z: ret where: ret = x+y ret -= z .x -> .y -> .z -> ret where: ret = x+y ret -= z as-with x def as y def as z def ret where: ret = x+y ret -= z # arrow transform (to avoid endless parentheses and try to be more readable. >> range(5) -> map(.x->x+2, _) -> list(_) >> [2,3,4,5,6] # pattern matching # use "condic" as keyword is for avoiding the conflictions against the standard libraries and packages from third party. "switch" and "match" both lead to conflictions. condic+(type) 1: case a:int => assert a == 1 and type(a) == 1 [>] case 0 => assert 1 > 0 [is not] case 1 => assert 1 is not 1 otherwise => print("nothing") condic+() [1,2,3]: case (a,*b)->b:list => sum(b) +[] case [] => print('empty list') +[==] case (a,b):(1,2) => print("the list is [1,2]") The grammars with more details and examples can be found in https://github.com/thautwarm/flowpython/wiki Does it interest you? If so, you can try it if you have CPython 3.6.x. pip install flowpython python -m flowpython -m enable/disable Here is an example to use flowpython, which gives the permutations of a sequence. from copy import deepcopy permutations = .seq -> seq_seq where: condic+[] seq: case (a, ) => seq_seq = [a,] case (a, b) => seq_seq = [[a,b],[b,a]] case (a,*b) => seq_seq = permutations(b) -> map(.x -> insertAll(x, a), _) -> sum(_, []) where: insertAll = . x, a -> ret where: ret = [ deepcopy(x) -> _.insert(i, a) or _ for i in (len(x) -> range(_+1)) ] If the object permutations are defined, try these codes in console: >> range(3) -> permutations(_) >> [[0, 1, 2], [1, 0, 2], [1, 2, 0], [0, 2, 1], [2, 0, 1], [2, 1, 0]] Does it seem to be interesting? Thanks, Thautwarm -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 2 Date: Sun, 13 Aug 2017 14:44:24 -0400 From: Yury Selivanov To: Nick Coghlan Cc: Nathaniel Smith , "python-ideas(a)python.org" Subject: Re: [Python-ideas] New PEP 550: Execution Context Message-ID: Content-Type: text/plain; charset="UTF-8" I'll start a new thread to discuss is we want this specific semantics change soon (with some updates). Yury ------------------------------ Message: 3 Date: Sun, 13 Aug 2017 12:14:07 -0700 From: Nathaniel Smith To: Yury Selivanov Cc: Nick Coghlan , Python-Ideas Subject: Re: [Python-ideas] New PEP 550: Execution Context Message-ID: Content-Type: text/plain; charset="UTF-8" On Sun, Aug 13, 2017 at 9:57 AM, Yury Selivanov wrote: > 2. ContextItem.has(), ContextItem.get(), ContextItem.set(), > ContextItem.delete() -- pretty self-explanatory. It might make sense to simplify even further and declare that context items are initialized to None to start, and the only operations are set() and get(). And then get() can't fail, b/c there is no "value missing" state. -n -- Nathaniel J. Smith -- https://vorpus.org ------------------------------ Subject: Digest Footer _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas ------------------------------ End of Python-ideas Digest, Vol 129, Issue 44 ********************************************* [View Less]

1 0

New PEP 550: Execution Context
by Stefan Krah Aug. 13, 2017

Aug. 13, 2017

Yury Selivanov wrote: > This is a new PEP to implement Execution Contexts in Python. The idea is of course great! A couple of issues for decimal: > Moreover, passing the context explicitly does not work at all for > libraries like ``decimal`` or ``numpy``, which use operator overloading. Instead of "with localcontext() ...", each coroutine can create a new Context() and use its methods, without any loss of functionality. All one loses is the inline operator syntax sugar. I'm … [View More]

2 1

Re: [Python-ideas] New PEP 550: Execution Context
by rymg19＠gmail.com Aug. 12, 2017

Aug. 12, 2017

So, I'm hardly an expert when it comes to things like this, but there are two things about this that don't seem right to me. (Also, I'd love to respond inline, but that's kind of difficult from a mobile phone.) The first is how set/get_execution_context_item take strings. Inevitably, people are going to do things like: CONTEXT_ITEM_NAME = 'foo-bar' ... sys.set_execution_context_item(CONTEXT_ITEM_NAME, 'stuff') IMO it would be nicer if there could be a key object used instead, e.g. my_key = … [View More]sys.execution_context_key('name-here-for-debugging-purposes') sys.set_execution_context_item(my_key, 'stuff') The advantage here would be no need for string constants and no potential naming conflicts (the string passed to the key creator would be used just for debugging, kind of like Thread names). Second thing is this: def context(x): old_x = get_execution_context_item('x') set_execution_context_item('x', x) try: yield finally: set_execution_context_item('x', old_x) If this would be done frequently, a context manager would be a *lot* more Pythonic, e.g.: with sys.temp_change_execution_context('x', new_x): # ... -- Ryan (ライアン) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone elsehttp://refi64.com On Aug 11, 2017 at 5:38 PM, <Yury Selivanov <yselivanov.ml(a)gmail.com>> wrote: Hi, This is a new PEP to implement Execution Contexts in Python. The PEP is in-flight to python.org, and in the meanwhile can be read on GitHub: https://github.com/python/peps/blob/master/pep-0550.rst (it contains a few diagrams and charts, so please read it there.) Thank you! Yury PEP: 550 Title: Execution Context Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov <yury(a)magic.io> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-Aug-2017 Python-Version: 3.7 Post-History: 11-Aug-2017 Abstract ======== This PEP proposes a new mechanism to manage execution state--the logical environment in which a function, a thread, a generator, or a coroutine executes in. A few examples of where having a reliable state storage is required: * Context managers like decimal contexts, ``numpy.errstate``, and ``warnings.catch_warnings``; * Storing request-related data such as security tokens and request data in web applications; * Profiling, tracing, and logging in complex and large code bases. The usual solution for storing state is to use a Thread-local Storage (TLS), implemented in the standard library as ``threading.local()``. Unfortunately, TLS does not work for isolating state of generators or asynchronous code because such code shares a single thread. Rationale ========= Traditionally a Thread-local Storage (TLS) is used for storing the state. However, the major flaw of using the TLS is that it works only for multi-threaded code. It is not possible to reliably contain the state within a generator or a coroutine. For example, consider the following generator:: def calculate(precision, ...): with decimal.localcontext() as ctx: # Set the precision for decimal calculations # inside this block ctx.prec = precision yield calculate_something() yield calculate_something_else() Decimal context is using a TLS to store the state, and because TLS is not aware of generators, the state can leak. The above code will not work correctly, if a user iterates over the ``calculate()`` generator with different precisions in parallel:: g1 = calculate(100) g2 = calculate(50) items = list(zip(g1, g2)) # items[0] will be a tuple of: # first value from g1 calculated with 100 precision, # first value from g2 calculated with 50 precision. # # items[1] will be a tuple of: # second value from g1 calculated with 50 precision, # second value from g2 calculated with 50 precision. An even scarier example would be using decimals to represent money in an async/await application: decimal calculations can suddenly lose precision in the middle of processing a request. Currently, bugs like this are extremely hard to find and fix. Another common need for web applications is to have access to the current request object, or security context, or, simply, the request URL for logging or submitting performance tracing data:: async def handle_http_request(request): context.current_http_request = request await ... # Invoke your framework code, render templates, # make DB queries, etc, and use the global # 'current_http_request' in that code. # This isn't currently possible to do reliably # in asyncio out of the box. These examples are just a few out of many, where a reliable way to store context data is absolutely needed. The inability to use TLS for asynchronous code has lead to proliferation of ad-hoc solutions, limited to be supported only by code that was explicitly enabled to work with them. Current status quo is that any library, including the standard library, that uses a TLS, will likely not work as expected in asynchronous code or with generators (see [3]_ as an example issue.) Some languages that have coroutines or generators recommend to manually pass a ``context`` object to every function, see [1]_ describing the pattern for Go. This approach, however, has limited use for Python, where we have a huge ecosystem that was built to work with a TLS-like context. Moreover, passing the context explicitly does not work at all for libraries like ``decimal`` or ``numpy``, which use operator overloading. .NET runtime, which has support for async/await, has a generic solution of this problem, called ``ExecutionContext`` (see [2]_). On the surface, working with it is very similar to working with a TLS, but the former explicitly supports asynchronous code. Goals ===== The goal of this PEP is to provide a more reliable alternative to ``threading.local()``. It should be explicitly designed to work with Python execution model, equally supporting threads, generators, and coroutines. An acceptable solution for Python should meet the following requirements: * Transparent support for code executing in threads, coroutines, and generators with an easy to use API. * Negligible impact on the performance of the existing code or the code that will be using the new mechanism. * Fast C API for packages like ``decimal`` and ``numpy``. Explicit is still better than implicit, hence the new APIs should only be used when there is no option to pass the state explicitly. With this PEP implemented, it should be possible to update a context manager like the below:: _local = threading.local() @contextmanager def context(x): old_x = getattr(_local, 'x', None) _local.x = x try: yield finally: _local.x = old_x to a more robust version that can be reliably used in generators and async/await code, with a simple transformation:: @contextmanager def context(x): old_x = get_execution_context_item('x') set_execution_context_item('x', x) try: yield finally: set_execution_context_item('x', old_x) Specification ============= This proposal introduces a new concept called Execution Context (EC), along with a set of Python APIs and C APIs to interact with it. EC is implemented using an immutable mapping. Every modification of the mapping produces a new copy of it. To illustrate what it means let's compare it to how we work with tuples in Python:: a0 = () a1 = a0 + (1,) a2 = a1 + (2,) # a0 is an empty tuple # a1 is (1,) # a2 is (1, 2) Manipulating an EC object would be similar:: a0 = EC() a1 = a0.set('foo', 'bar') a2 = a1.set('spam', 'ham') # a0 is an empty mapping # a1 is {'foo': 'bar'} # a2 is {'foo': 'bar', 'spam': 'ham'} In CPython, every thread that can execute Python code has a corresponding ``PyThreadState`` object. It encapsulates important runtime information like a pointer to the current frame, and is being used by the ceval loop extensively. We add a new field to ``PyThreadState``, called ``exec_context``, which points to the current EC object. We also introduce a set of APIs to work with Execution Context. In this section we will only cover two functions that are needed to explain how Execution Context works. See the full list of new APIs in the `New APIs`_ section. * ``sys.get_execution_context_item(key, default=None)``: lookup ``key`` in the EC of the executing thread. If not found, return ``default``. * ``sys.set_execution_context_item(key, value)``: get the current EC of the executing thread. Add a ``key``/``value`` item to it, which will produce a new EC object. Set the new object as the current one for the executing thread. In pseudo-code:: tstate = PyThreadState_GET() ec = tstate.exec_context ec2 = ec.set(key, value) tstate.exec_context = ec2 Note, that some important implementation details and optimizations are omitted here, and will be covered in later sections of this PEP. Now let's see how Execution Contexts work with regular multi-threaded code, generators, and coroutines. Regular & Multithreaded Code ---------------------------- For regular Python code, EC behaves just like a thread-local. Any modification of the EC object produces a new one, which is immediately set as the current one for the thread state. .. figure:: pep-0550/functions.png :align: center :width: 90% Figure 1. Execution Context flow in a thread. As Figure 1 illustrates, if a function calls ``set_execution_context_item()``, the modification of the execution context will be visible to all subsequent calls and to the caller:: def set_foo(): set_execution_context_item('foo', 'spam') set_execution_context_item('foo', 'bar') print(get_execution_context_item('foo')) set_foo() print(get_execution_context_item('foo')) # will print: # bar # spam Coroutines ---------- Python :pep:`492` coroutines are used to implement cooperative multitasking. For a Python end-user they are similar to threads, especially when it comes to sharing resources or modifying the global state. An event loop is needed to schedule coroutines. Coroutines that are explicitly scheduled by the user are usually called Tasks. When a coroutine is scheduled, it can schedule other coroutines using an ``await`` expression. In async/await world, awaiting a coroutine can be viewed as a different calling convention: Tasks are similar to threads, and awaiting on coroutines within a Task is similar to calling functions within a thread. By drawing a parallel between regular multithreaded code and async/await, it becomes apparent that any modification of the execution context within one Task should be visible to all coroutines scheduled within it. Any execution context modifications, however, must not be visible to other Tasks executing within the same thread. To achieve this, a small set of modifications to the coroutine object is needed: * When a coroutine object is instantiated, it saves a reference to the current execution context object to its ``cr_execution_context`` attribute. * Coroutine's ``.send()`` and ``.throw()`` methods are modified as follows (in pseudo-C):: if coro->cr_isolated_execution_context: # Save a reference to the current execution context old_context = tstate->execution_context # Set our saved execution context as the current # for the current thread. tstate->execution_context = coro->cr_execution_context try: # Perform the actual `Coroutine.send()` or # `Coroutine.throw()` call. return coro->send(...) finally: # Save a reference to the updated execution_context. # We will need it later, when `.send()` or `.throw()` # are called again. coro->cr_execution_context = tstate->execution_context # Restore thread's execution context to what it was before # invoking this coroutine. tstate->execution_context = old_context else: # Perform the actual `Coroutine.send()` or # `Coroutine.throw()` call. return coro->send(...) * ``cr_isolated_execution_context`` is a new attribute on coroutine objects. Set to ``True`` by default, it makes any execution context modifications performed by coroutine to stay visible only to that coroutine. When Python interpreter sees an ``await`` instruction, it flips ``cr_isolated_execution_context`` to ``False`` for the coroutine that is about to be awaited. This makes any changes to execution context made by nested coroutine calls within a Task to be visible throughout the Task. Because the top-level coroutine (Task) cannot be scheduled with ``await`` (in asyncio you need to call ``loop.create_task()`` or ``asyncio.ensure_future()`` to schedule a Task), all execution context modifications are guaranteed to stay within the Task. * We always work with ``tstate->exec_context``. We use ``coro->cr_execution_context`` only to store coroutine's execution context when it is not executing. Figure 2 below illustrates how execution context mutations work with coroutines. .. figure:: pep-0550/coroutines.png :align: center :width: 90% Figure 2. Execution Context flow in coroutines. In the above diagram: * When "coro1" is created, it saves a reference to the current execution context "2". * If it makes any change to the context, it will have its own execution context branch "2.1". * When it awaits on "coro2", any subsequent changes it does to the execution context are visible to "coro1", but not outside of it. In code:: async def inner_foo(): print('inner_foo:', get_execution_context_item('key')) set_execution_context_item('key', 2) async def foo(): print('foo:', get_execution_context_item('key')) set_execution_context_item('key', 1) await inner_foo() print('foo:', get_execution_context_item('key')) set_execution_context_item('key', 'spam') print('main:', get_execution_context_item('key')) asyncio.get_event_loop().run_until_complete(foo()) print('main:', get_execution_context_item('key')) which will output:: main: spam foo: spam inner_foo: 1 foo: 2 main: spam Generator-based coroutines (generators decorated with ``types.coroutine`` or ``asyncio.coroutine``) behave exactly as native coroutines with regards to execution context management: their ``yield from`` expression is semantically equivalent to ``await``. Generators ---------- Generators in Python, while similar to Coroutines, are used in a fundamentally different way. They are producers of data, and they use ``yield`` expression to suspend/resume their execution. A crucial difference between ``await coro`` and ``yield value`` is that the former expression guarantees that the ``coro`` will be executed to the end, while the latter is producing ``value`` and suspending the generator until it gets iterated again. Generators share 99% of their implementation with coroutines, and thus have similar new attributes ``gi_execution_context`` and ``gi_isolated_execution_context``. Similar to coroutines, generators save a reference to the current execution context when they are instantiated. The have the same implementation of ``.send()`` and ``.throw()`` methods. The only difference is that ``gi_isolated_execution_context`` is always set to ``True``, and is never modified by the interpreter. ``yield from o`` expression in regular generators that are not decorated with ``types.coroutine``, is semantically equivalent to ``for v in o: yield v``. .. figure:: pep-0550/generators.png :align: center :width: 90% Figure 3. Execution Context flow in a generator. In the above diagram: * When "gen1" is created, it saves a reference to the current execution context "2". * If it makes any change to the context, it will have its own execution context branch "2.1". * When "gen2" is created, it saves a reference to the current execution context for it -- "2.1". * Any subsequent execution context updated in "gen2" will only be visible to "gen2". * Likewise, any context changes that "gen1" will do after it created "gen2" will not be visible to "gen2". In code:: def inner_foo(): for i in range(3): print('inner_foo:', get_execution_context_item('key')) set_execution_context_item('key', i) yield i def foo(): set_execution_context_item('key', 'spam') print('foo:', get_execution_context_item('key')) inner = inner_foo() while True: val = next(inner, None) if val is None: break yield val print('foo:', get_execution_context_item('key')) set_execution_context_item('key', 'spam') print('main:', get_execution_context_item('key')) list(foo()) print('main:', get_execution_context_item('key')) which will output:: main: ham foo: spam inner_foo: spam foo: spam inner_foo: 0 foo: spam inner_foo: 1 foo: spam main: ham As we see, any modification of the execution context in a generator is visible only to the generator itself. There is one use-case where it is desired for generators to affect the surrounding execution context: ``contextlib.contextmanager`` decorator. To make the following work:: @contextmanager def context(x): old_x = get_execution_context_item('x') set_execution_context_item('x', x) try: yield finally: set_execution_context_item('x', old_x) we modified ``contextmanager`` to flip ``gi_isolated_execution_context`` flag to ``False`` on its generator. Greenlets --------- Greenlet is an alternative implementation of cooperative scheduling for Python. Although greenlet package is not part of CPython, popular frameworks like gevent rely on it, and it is important that greenlet can be modified to support execution contexts. In a nutshell, greenlet design is very similar to design of generators. The main difference is that for generators, the stack is managed by the Python interpreter. Greenlet works outside of the Python interpreter, and manually saves some ``PyThreadState`` fields and pushes/pops the C-stack. Since Execution Context is implemented on top of ``PyThreadState``, it's easy to add transparent support of it to greenlet. New APIs ======== Even though this PEP adds a number of new APIs, please keep in mind, that most Python users will likely ever use only two of them: ``sys.get_execution_context_item()`` and ``sys.set_execution_context_item()``. Python ------ 1. ``sys.get_execution_context_item(key, default=None)``: lookup ``key`` for the current Execution Context. If not found, return ``default``. 2. ``sys.set_execution_context_item(key, value)``: set ``key``/``value`` item for the current Execution Context. If ``value`` is ``None``, the item will be removed. 3. ``sys.get_execution_context()``: return the current Execution Context object: ``sys.ExecutionContext``. 4. ``sys.set_execution_context(ec)``: set the passed ``sys.ExecutionContext`` instance as a current one for the current thread. 5. ``sys.ExecutionContext`` object. Implementation detail: ``sys.ExecutionContext`` wraps a low-level ``PyExecContextData`` object. ``sys.ExecutionContext`` has a mutable mapping API, abstracting away the real immutable ``PyExecContextData``. * ``ExecutionContext()``: construct a new, empty, execution context. * ``ec.run(func, *args)`` method: run ``func(*args)`` in the ``ec`` execution context. * ``ec[key]``: lookup ``key`` in ``ec`` context. * ``ec[key] = value``: assign ``key``/``value`` item to the ``ec``. * ``ec.get()``, ``ec.items()``, ``ec.values()``, ``ec.keys()``, and ``ec.copy()`` are similar to that of ``dict`` object. C API ----- C API is different from the Python one because it operates directly on the low-level immutable ``PyExecContextData`` object. 1. New ``PyThreadState->exec_context`` field, pointing to a ``PyExecContextData`` object. 2. ``PyThreadState_SetExecContextItem`` and ``PyThreadState_GetExecContextItem`` similar to ``sys.set_execution_context_item()`` and ``sys.get_execution_context_item()``. 3. ``PyThreadState_GetExecContext``: similar to ``sys.get_execution_context()``. Always returns an ``PyExecContextData`` object. If ``PyThreadState->exec_context`` is ``NULL`` an new and empty one will be created and assigned to ``PyThreadState->exec_context``. 4. ``PyThreadState_SetExecContext``: similar to ``sys.set_execution_context()``. 5. ``PyExecContext_New``: create a new empty ``PyExecContextData`` object. 6. ``PyExecContext_SetItem`` and ``PyExecContext_GetItem``. The exact layout ``PyExecContextData`` is private, which allows to switch it to a different implementation later. More on that in the `Implementation Details`_ section. Modifications in Standard Library ================================= * ``contextlib.contextmanager`` was updated to flip the new ``gi_isolated_execution_context`` attribute on the generator. * ``asyncio.events.Handle`` object now captures the current execution context when it is created, and uses the saved execution context to run the callback (with ``ExecutionContext.run()`` method.) This makes ``loop.call_soon()`` to run callbacks in the execution context they were scheduled. No modifications in ``asyncio.Task`` or ``asyncio.Future`` were necessary. Some standard library modules like ``warnings`` and ``decimal`` can be updated to use new execution contexts. This will be considered in separate issues if this PEP is accepted. Backwards Compatibility ======================= This proposal preserves 100% backwards compatibility. Performance =========== Implementation Details ---------------------- The new ``PyExecContextData`` object is wrapping a ``dict`` object. Any modification requires creating a shallow copy of the dict. While working on the reference implementation of this PEP, we were able to optimize ``dict.copy()`` operation **5.5x**, see [4]_ for details. .. figure:: pep-0550/dict_copy.png :align: center :width: 100% Figure 4. Figure 4 shows that the performance of immutable dict implemented with shallow copying is expectedly O(n) for the ``set()`` operation. However, this is tolerable until dict has more than 100 items (1 ``set()`` takes about a microsecond.) Judging by the number of modules that need EC in Standard Library it is likely that real world Python applications will use significantly less than 100 execution context variables. The important point is that the cost of accessing a key in Execution Context is always O(1). If the ``set()`` operation performance is a major concern, we discuss alternative approaches that have O(1) or close ``set()`` performance in `Alternative Immutable Dict Implementation`_, `Faster C API`_, and `Copy-on-write Execution Context`_ sections. Generators and Coroutines ------------------------- Using a microbenchmark for generators and coroutines from :pep:`492` ([12]_), it was possible to observe 0.5 to 1% performance degradation. asyncio echoserver microbechmarks from the uvloop project [13]_ showed 1-1.5% performance degradation for asyncio code. asyncpg benchmarks [14]_, that execute more code and are closer to a real-world application did not exhibit any noticeable performance change. Overall Performance Impact -------------------------- The total number of changed lines in the ceval loop is 2 -- in the ``YIELD_FROM`` opcode implementation. Only performance of generators and coroutines can be affected by the proposal. This was confirmed by running Python Performance Benchmark Suite [15]_, which demonstrated that there is no difference between 3.7 master branch and this PEP reference implementation branch (full benchmark results can be found here [16]_.) Design Considerations ===================== Alternative Immutable Dict Implementation ----------------------------------------- Languages like Clojure and Scala use Hash Array Mapped Tries (HAMT) to implement high performance immutable collections [5]_, [6]_. Immutable mappings implemented with HAMT have O(log\ :sub:`32`\ N) performance for both ``set()`` and ``get()`` operations, which will be essentially O(1) for relatively small mappings in EC. To assess if HAMT can be used for Execution Context, we implemented it in CPython [7]_. .. figure:: pep-0550/hamt_vs_dict.png :align: center :width: 100% Figure 5. Benchmark code can be found here: [9]_. Figure 5 shows that HAMT indeed displays O(1) performance for all benchmarked dictionary sizes. For dictionaries with less than 100 items, HAMT is a bit slower than Python dict/shallow copy. .. figure:: pep-0550/lookup_hamt.png :align: center :width: 100% Figure 6. Benchmark code can be found here: [10]_. Figure 6 below shows comparison of lookup costs between Python dict and an HAMT immutable mapping. HAMT lookup time is 30-40% worse than Python dict lookups on average, which is a very good result, considering how well Python dicts are optimized. Note, that according to [8]_, HAMT design can be further improved. The bottom line is that the current approach with implementing an immutable mapping with shallow-copying dict will likely perform adequately in real-life applications. The HAMT solution is more future proof, however. The proposed API is designed in such a way that the underlying implementation of the mapping can be changed completely without affecting the Execution Context `Specification`_, which allows us to switch to HAMT at some point if necessary. Copy-on-write Execution Context ------------------------------- The implementation of Execution Context in .NET is different from this PEP. .NET uses copy-on-write mechanism and a regular mutable mapping. One way to implement this in CPython would be to have two new fields in ``PyThreadState``: * ``exec_context`` pointing to the current Execution Context mapping; * ``exec_context_copy_on_write`` flag, set to ``0`` initially. The idea is that whenever we are modifying the EC, the copy-on-write flag is checked, and if it is set to ``1``, the EC is copied. Modifications to Coroutine and Generator ``.send()`` and ``.throw()`` methods described in the `Coroutines`_ section will be almost the same, except that in addition to the ``gi_execution_context`` they will have a ``gi_exec_context_copy_on_write`` flag. When a coroutine or a generator starts, the flag will be set to ``1``. This will ensure that any modification of the EC performed within a coroutine or a generator will be isolated. This approach has one advantage: * For Execution Context that contains a large number of items, copy-on-write is a more efficient solution than the shallow-copy dict approach. However, we believe that copy-on-write disadvantages are more important to consider: * Copy-on-write behaviour for generators and coroutines makes EC semantics less predictable. With immutable EC approach, generators and coroutines always execute in the EC that was current at the moment of their creation. Any modifications to the outer EC while a generator or a coroutine is executing are not visible to them:: def generator(): yield 1 print(get_execution_context_item('key')) yield 2 set_execution_context_item('key', 'spam') gen = iter(generator()) next(gen) set_execution_context_item('key', 'ham') next(gen) The above script will always print 'spam' with immutable EC. With a copy-on-write approach, the above script will print 'ham'. Now, consider that ``generator()`` was refactored to call some library function, that uses Execution Context:: def generator(): yield 1 some_function_that_uses_decimal_context() print(get_execution_context_item('key')) yield 2 Now, the script will print 'spam', because ``some_function_that_uses_decimal_context`` forced the EC to copy, and ``set_execution_context_item('key', 'ham')`` line did not affect the ``generator()`` code after all. * Similarly to the previous point, ``sys.ExecutionContext.run()`` method will also become less predictable, as ``sys.get_execution_context()`` would still return a reference to the current mutable EC. We can't modify ``sys.get_execution_context()`` to return a shallow copy of the current EC, because this would seriously harm performance of ``asyncio.call_soon()`` and similar places, where it is important to propagate the Execution Context. * Even though copy-on-write requires to shallow copy the execution context object less frequently, copying will still take place in coroutines and generators. In which case, HAMT approach will perform better for medium to large sized execution contexts. All in all, we believe that the copy-on-write approach introduces very subtle corner cases that could lead to bugs that are exceptionally hard to discover and fix. The immutable EC solution in comparison is always predictable and easy to reason about. Therefore we believe that any slight performance gain that the copy-on-write solution might offer is not worth it. Faster C API ------------ Packages like numpy and standard library modules like decimal need to frequently query the global state for some local context configuration. It is important that the APIs that they use is as fast as possible. The proposed ``PyThreadState_SetExecContextItem`` and ``PyThreadState_GetExecContextItem`` functions need to get the current thread state with ``PyThreadState_GET()`` (fast) and then perform a hash lookup (relatively slow). We can eliminate the hash lookup by adding three additional C API functions: * ``Py_ssize_t PyExecContext_RequestIndex(char *key_name)``: a function similar to the existing ``_PyEval_RequestCodeExtraIndex`` introduced :pep:`523`. The idea is to request a unique index that can later be used to lookup context items. The ``key_name`` can later be used by ``sys.ExecutionContext`` to introspect items added with this API. * ``PyThreadState_SetExecContextIndexedItem(Py_ssize_t index, PyObject *val)`` and ``PyThreadState_GetExecContextIndexedItem(Py_ssize_t index)`` to request an item by its index, avoiding the cost of hash lookup. Why setting a key to None removes the item? ------------------------------------------- Consider a context manager:: @contextmanager def context(x): old_x = get_execution_context_item('x') set_execution_context_item('x', x) try: yield finally: set_execution_context_item('x', old_x) With ``set_execution_context_item(key, None)`` call removing the ``key``, the user doesn't need to write additional code to remove the ``key`` if it wasn't in the execution context already. An alternative design with ``del_execution_context_item()`` method would look like the following:: @contextmanager def context(x): not_there = object() old_x = get_execution_context_item('x', not_there) set_execution_context_item('x', x) try: yield finally: if old_x is not_there: del_execution_context_item('x') else: set_execution_context_item('x', old_x) Can we fix ``PyThreadState_GetDict()``? --------------------------------------- ``PyThreadState_GetDict`` is a TLS, and some of its existing users might depend on it being just a TLS. Changing its behaviour to follow the Execution Context semantics would break backwards compatibility. PEP 521 ------- :pep:`521` proposes an alternative solution to the problem: enhance Context Manager Protocol with two new methods: ``__suspend__`` and ``__resume__``. To make it compatible with async/await, the Asynchronous Context Manager Protocol will also need to be extended with ``__asuspend__`` and ``__aresume__``. This allows to implement context managers like decimal context and ``numpy.errstate`` for generators and coroutines. The following code:: class Context: def __enter__(self): self.old_x = get_execution_context_item('x') set_execution_context_item('x', 'something') def __exit__(self, *err): set_execution_context_item('x', self.old_x) would become this:: class Context: def __enter__(self): self.old_x = get_execution_context_item('x') set_execution_context_item('x', 'something') def __suspend__(self): set_execution_context_item('x', self.old_x) def __resume__(self): set_execution_context_item('x', 'something') def __exit__(self, *err): set_execution_context_item('x', self.old_x) Besides complicating the protocol, the implementation will likely negatively impact performance of coroutines, generators, and any code that uses context managers, and will notably complicate the interpreter implementation. It also does not solve the leaking state problem for greenlet/gevent. :pep:`521` also does not provide any mechanism to propagate state in a local context, like storing a request object in an HTTP request handler to have better logging. Can Execution Context be implemented outside of CPython? -------------------------------------------------------- Because async/await code needs an event loop to run it, an EC-like solution can be implemented in a limited way for coroutines. Generators, on the other hand, do not have an event loop or trampoline, making it impossible to intercept their ``yield`` points outside of the Python interpreter. Reference Implementation ======================== The reference implementation can be found here: [11]_. References ========== .. [1] https://blog.golang.org/context .. [2] https://msdn.microsoft.com/en-us/library/system.threading.executioncontext.… .. [3] https://github.com/numpy/numpy/issues/9444 .. [4] http://bugs.python.org/issue31179 .. [5] https://en.wikipedia.org/wiki/Hash_array_mapped_trie .. [6] http://blog.higher-order.net/2010/08/16/assoc-and-clojures-persistenthashma… .. [7] https://github.com/1st1/cpython/tree/hamt .. [8] https://michael.steindorfer.name/publications/oopsla15.pdf .. [9] https://gist.github.com/1st1/9004813d5576c96529527d44c5457dcd .. [10] https://gist.github.com/1st1/dbe27f2e14c30cce6f0b5fddfc8c437e .. [11] https://github.com/1st1/cpython/tree/pep550 .. [12] https://www.python.org/dev/peps/pep-0492/#async-await .. [13] https://github.com/MagicStack/uvloop/blob/master/examples/bench/echoserver.… .. [14] https://github.com/MagicStack/pgbench .. [15] https://github.com/python/performance .. [16] https://gist.github.com/1st1/6b7a614643f91ead3edf37c4451a6b4c Copyright ========= This document has been placed in the public domain. _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ [View Less]

2 1