Mailman 3 October 2017 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists … [View More]now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich [View Less]

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann Feb. 22, 2020

Feb. 22, 2020

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int … [View More]

14 20

Asynchronous exception handling around with/try statement borders
by Erik Bray Sept. 24, 2018

Sept. 24, 2018

Hi folks, I normally wouldn't bring something like this up here, except I think that there is possibility of something to be done--a language documentation clarification if nothing else, though possibly an actual code change as well. I've been having an argument with a colleague over the last couple days over the proper way order of statements when setting up a try/finally to perform cleanup of some action. On some level we're both being stubborn I think, and I'm not looking for resolution … [View More]as to who's right/wrong or I wouldn't bring it to this list in the first place. The original argument was over setting and later restoring os.environ, but we ended up arguing over threading.Lock.acquire/release which I think is a more interesting example of the problem, and he did raise a good point that I do want to bring up. </prologue> My colleague's contention is that given lock = threading.Lock() this is simply *wrong*: lock.acquire() try: do_something() finally: lock.release() whereas this is okay: with lock: do_something() Ignoring other details of how threading.Lock is actually implemented, assuming that Lock.__enter__ calls acquire() and Lock.__exit__ calls release() then as far as I've known ever since Python 2.5 first came out these two examples are semantically *equivalent*, and I can't find any way of reading PEP 343 or the Python language reference that would suggest otherwise. However, there *is* a difference, and has to do with how signals are handled, particularly w.r.t. context managers implemented in C (hence we are talking CPython specifically): If Lock.__enter__ is a pure Python method (even if it maybe calls some C methods), and a SIGINT is handled during execution of that method, then in almost all cases a KeyboardInterrupt exception will be raised from within Lock.__enter__--this means the suite under the with: statement is never evaluated, and Lock.__exit__ is never called. You can be fairly sure the KeyboardInterrupt will be raised from somewhere within a pure Python Lock.__enter__ because there will usually be at least one remaining opcode to be evaluated, such as RETURN_VALUE. Because of how delayed execution of signal handlers is implemented in the pyeval main loop, this means the signal handler for SIGINT will be called *before* RETURN_VALUE, resulting in the KeyboardInterrupt exception being raised. Standard stuff. However, if Lock.__enter__ is a PyCFunction things are quite different. If you look at how the SETUP_WITH opcode is implemented, it first calls the __enter__ method with _PyObjet_CallNoArg. If this returns NULL (i.e. an exception occurred in __enter__) then "goto error" is executed and the exception is raised. However if it returns non-NULL the finally block is set up with PyFrame_BlockSetup and execution proceeds to the next opcode. At this point a potentially waiting SIGINT is handled, resulting in KeyboardInterrupt being raised while inside the with statement's suite, and finally block, and hence Lock.__exit__ are entered. Long story short, because Lock.__enter__ is a C function, assuming that it succeeds normally then with lock: do_something() always guarantees that Lock.__exit__ will be called if a SIGINT was handled inside Lock.__enter__, whereas with lock.acquire() try: ... finally: lock.release() there is at last a small possibility that the SIGINT handler is called after the CALL_FUNCTION op but before the try/finally block is entered (e.g. before executing POP_TOP or SETUP_FINALLY). So the end result is that the lock is held and never released after the KeyboardInterrupt (whether or not it's handled somehow). Whereas, again, if Lock.__enter__ is a pure Python function there's less likely to be any difference (though I don't think the possibility can be ruled out entirely). At the very least I think this quirk of CPython should be mentioned somewhere (since in all other cases the semantic meaning of the "with:" statement is clear). However, I think it might be possible to gain more consistency between these cases if pending signals are checked/handled after any direct call to PyCFunction from within the ceval loop. Sorry for the tl;dr; any thoughts? [View Less]

7 15

Positional-only parameters
by Victor Stinner Sept. 11, 2018

Sept. 11, 2018

Hi, For technical reasons, many functions of the Python standard libraries implemented in C have positional-only parameters. Example: ------- $ ./python Python 3.7.0a0 (default, Feb 25 2017, 04:30:32) >>> help(str.replace) replace(self, old, new, count=-1, /) # <== notice "/" at the end ... >>> "a".replace("x", "y") # ok 'a' >>> "a".replace(old="x", new="y") # ERR! TypeError: replace() takes at least 2 arguments (0 given) ------- When converting the … [View More]

19 27

Implicit string literal concatenation considered harmful?
by Guido van Rossum March 14, 2018

March 14, 2018

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' … [View More]

51 165

PEP 560 (second post)
by Ivan Levkivskyi Nov. 15, 2017

Nov. 15, 2017

Previously I posted PEP 560 two weeks ago, while several other PEPs were also posted, so it didn't get much of attention. Here I post the PEP 560 again, now including the full text for convenience of commenting. -- Ivan ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ PEP: 560 Title: Core support for generic types Author: Ivan Levkivskyi <levkivskyi(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 03-Sep-2017 Python-Version: 3.7 … [View More]Post-History: 09-Sep-2017 Abstract ======== Initially PEP 484 was designed in such way that it would not introduce *any* changes to the core CPython interpreter. Now type hints and the ``typing`` module are extensively used by the community, e.g. PEP 526 and PEP 557 extend the usage of type hints, and the backport of ``typing`` on PyPI has 1M downloads/month. Therefore, this restriction can be removed. It is proposed to add two special methods ``__class_getitem__`` and ``__subclass_base__`` to the core CPython for better support of generic types. Rationale ========= The restriction to not modify the core CPython interpreter lead to some design decisions that became questionable when the ``typing`` module started to be widely used. There are three main points of concerns: performance of the ``typing`` module, metaclass conflicts, and the large number of hacks currently used in ``typing``. Performance: ------------ The ``typing`` module is one of the heaviest and slowest modules in the standard library even with all the optimizations made. Mainly this is because subscripted generic types (see PEP 484 for definition of terms used in this PEP) are class objects (see also [1]_). The three main ways how the performance can be improved with the help of the proposed special methods: - Creation of generic classes is slow since the ``GenericMeta.__new__`` is very slow; we will not need it anymore. - Very long MROs for generic classes will be twice shorter; they are present because we duplicate the ``collections.abc`` inheritance chain in ``typing``. - Time of instantiation of generic classes will be improved (this is minor however). Metaclass conflicts: -------------------- All generic types are instances of ``GenericMeta``, so if a user uses a custom metaclass, then it is hard to make a corresponding class generic. This is particularly hard for library classes that a user doesn't control. A workaround is to always mix-in ``GenericMeta``:: class AdHocMeta(GenericMeta, LibraryMeta): pass class UserClass(LibraryBase, Generic[T], metaclass=AdHocMeta): ... but this is not always practical or even possible. With the help of the proposed special attributes the ``GenericMeta`` metaclass will not be needed. Hacks and bugs that will be removed by this proposal: ----------------------------------------------------- - ``_generic_new`` hack that exists since ``__init__`` is not called on instances with a type differing form the type whose ``__new__`` was called, ``C[int]().__class__ is C``. - ``_next_in_mro`` speed hack will be not necessary since subscription will not create new classes. - Ugly ``sys._getframe`` hack, this one is particularly nasty, since it looks like we can't remove it without changes outside ``typing``. - Currently generics do dangerous things with private ABC caches to fix large memory consumption that grows at least as O(N\ :sup:`2`), see [2]_. This point is also important because it was recently proposed to re-implement ``ABCMeta`` in C. - Problems with sharing attributes between subscripted generics, see [3]_. Current solution already uses ``__getattr__`` and ``__setattr__``, but it is still incomplete, and solving this without the current proposal will be hard and will need ``__getattribute__``. - ``_no_slots_copy`` hack, where we clean-up the class dictionary on every subscription thus allowing generics with ``__slots__``. - General complexity of the ``typing`` module, the new proposal will not only allow to remove the above mentioned hacks/bugs, but also simplify the implementation, so that it will be easier to maintain. Specification ============= The idea of ``__class_getitem__`` is simple: it is an exact analog of ``__getitem__`` with an exception that it is called on a class that defines it, not on its instances, this allows us to avoid ``GenericMeta.__getitem__`` for things like ``Iterable[int]``. The ``__class_getitem__`` is automatically a class method and does not require ``@classmethod`` decorator (similar to ``__init_subclass__``) and is inherited like normal attributes. For example:: class MyList: def __getitem__(self, index): return index + 1 def __class_getitem__(cls, item): return f"{cls.__name__}[{item.__name__}]" class MyOtherList(MyList): pass assert MyList()[0] == 1 assert MyList[int] == "MyList[int]" assert MyOtherList()[0] == 1 assert MyOtherList[int] == "MyOtherList[int]" Note that this method is used as a fallback, so if a metaclass defines ``__getitem__``, then that will have the priority. If an object that is not a class object appears in the bases of a class definition, the ``__subclass_base__`` is searched on it. If found, it is called with the original tuple of bases as an argument. If the result of the call is not ``None``, then it is substituted instead of this object. Otherwise (if the result is ``None``), the base is just removed. This is necessary to avoid inconsistent MRO errors, that are currently prevented by manipulations in ``GenericMeta.__new__``. After creating the class, the original bases are saved in ``__orig_bases__`` (currently this is also done by the metaclass). NOTE: These two method names are reserved for exclusive use by the ``typing`` module and the generic types machinery, and any other use is strongly discouraged. The reference implementation (with tests) can be found in [4]_, the proposal was originally posted and discussed on the ``typing`` tracker, see [5]_. Backwards compatibility and impact on users who don't use ``typing``: ===================================================================== This proposal may break code that currently uses the names ``__class_getitem__`` and ``__subclass_base__``. This proposal will support almost complete backwards compatibility with the current public generic types API; moreover the ``typing`` module is still provisional. The only two exceptions are that currently ``issubclass(List[int], List)`` returns True, with this proposal it will raise ``TypeError``. Also ``issubclass(collections.abc.Iterable, typing.Iterable)`` will return ``False``, which is probably desirable, since currently we have a (virtual) inheritance cycle between these two classes. With the reference implementation I measured negligible performance effects (under 1% on a micro-benchmark) for regular (non-generic) classes. References ========== .. [1] Discussion following Mark Shannon's presentation at Language Summit (https://github.com/python/typing/issues/432) .. [2] Pull Request to implement shared generic ABC caches (https://github.com/python/typing/pull/383) .. [3] An old bug with setting/accessing attributes on generic types (https://github.com/python/typing/issues/392) .. [4] The reference implementation (https://github.com/ilevkivskyi/cpython/pull/2/files) .. [5] Original proposal (https://github.com/python/typing/issues/468) Copyright ========= This document has been placed in the public domain. [View Less]

6 18

PEP 562
by Ivan Levkivskyi Nov. 14, 2017

Nov. 14, 2017

I have written a short PEP as a complement/alternative to PEP 549. I will be grateful for comments and suggestions. The PEP should appear online soon. -- Ivan *********************************************************** PEP: 562 Title: Module __getattr__ Author: Ivan Levkivskyi <levkivskyi(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 09-Sep-2017 Python-Version: 3.7 Post-History: 09-Sep-2017 Abstract ======== It is proposed to support ``… [View More]__getattr__`` function defined on modules to provide basic customization of module attribute access. Rationale ========= It is sometimes convenient to customize or otherwise have control over access to module attributes. A typical example is managing deprecation warnings. Typical workarounds are assigning ``__class__`` of a module object to a custom subclass of ``types.ModuleType`` or substituting ``sys.modules`` item with a custom wrapper instance. It would be convenient to simplify this procedure by recognizing ``__getattr__`` defined directly in a module that would act like a normal ``__getattr__`` method, except that it will be defined on module *instances*. For example:: # lib.py from warnings import warn deprecated_names = ["old_function", ...] def _deprecated_old_function(arg, other): ... def __getattr__(name): if name in deprecated_names: warn(f"{name} is deprecated", DeprecationWarning) return globals()[f"_deprecated_{name}"] raise AttributeError(f"module {__name__} has no attribute {name}") # main.py from lib import old_function # Works, but emits the warning There is a related proposal PEP 549 that proposes to support instance properties for a similar functionality. The difference is this PEP proposes a faster and simpler mechanism, but provides more basic customization. An additional motivation for this proposal is that PEP 484 already defines the use of module ``__getattr__`` for this purpose in Python stub files, see [1]_. Specification ============= The ``__getattr__`` function at the module level should accept one argument which is a name of an attribute and return the computed value or raise an ``AttributeError``:: def __getattr__(name: str) -> Any: ... This function will be called only if ``name`` is not found in the module through the normal attribute lookup. The reference implementation for this PEP can be found in [2]_. Backwards compatibility and impact on performance ================================================= This PEP may break code that uses module level (global) name ``__getattr__``. The performance implications of this PEP are minimal, since ``__getattr__`` is called only for missing attributes. References ========== .. [1] PEP 484 section about ``__getattr__`` in stub files (https://www.python.org/dev/peps/pep-0484/#stub-files) .. [2] The reference implementation (https://github.com/ilevkivskyi/cpython/pull/3/files) Copyright ========= This document has been placed in the public domain. [View Less]

10 17

install pip packages from Python prompt
by Stephan Houben Nov. 6, 2017

Nov. 6, 2017

Hi all, Here is in somewhat more detail my earlier proposal for having in the interactive Python interpreter a `pip` function to install packages from Pypi. Motivation: it appears to me that there is a category of newbies for which "open a shell and do `pip whatever`" is a bit too much. It would, in my opinion, simplify things a bit if they could just copy-and-paste some text into the Python interpreter and have some packages from pip installed. That would simplify instructions on how to … [View More]

18 52

Membership of infinite iterators
by Jason Campbell Nov. 3, 2017

Nov. 3, 2017

I recently came across a bug where checking negative membership (__contains__ returns False) of an infinite iterator will freeze the program. It may seem a bit obvious, but you can check membership in range for example without iterating over the entire range. `int(1e100) in range(int(1e101))` on my machine takes about 1.5us `int(1e7) in itertools.count()` on my machine takes about 250ms (and gets worse quite quickly). Any membership test on the infinite iterators that is negative will … [View More]

17 63

Re: [Python-ideas] Add single() to itertools
by Ivan Pozdeev Nov. 2, 2017

Nov. 2, 2017

On 30.10.2017 9:29, python-ideas-request(a)python.org wrote: > If I have understood your use-case, you have a function that returns a > list of results (or possibly an iterator, or a tuple, or some other > sequence): > > print(search(haystack, needle)) > # prints ['bronze needle', 'gold needle', 'silver needle'] > > There are times you expect there to be a single result, and if there are > multiple results, that is considered an error. Am I correct so far? … [View More]

9 11

Defining an easily installable "Recommended baseline package set"
by Nick Coghlan Nov. 2, 2017

Nov. 2, 2017

Over on python-dev, the question of recommending MRAB's "regex" module over the standard library's "re" module for more advanced regular expressions recently came up again. Because of various logistical issues and backwards compatibility risks, it's highly unlikely that we'll ever be able to swap out the current _sre based re module implementation in the standard library for an implementation based on the regex module. At the same time, it would be beneficial to have a way to offer an even … [View More]stronger recommendation to redistributors that we think full-featured general purpose Python scripting environments should offer the regex module as an opt-in alternative to the baseline re feature set, since that would also help with other currently difficult cases like the requests module. What I'm thinking is that we could make some relatively simple additions to the `ensurepip` and `venv` modules to help with this: 1. Add a ensurepip.RECOMMENDED_PACKAGES mapping keyed by standard library module names containing dependency specifiers for recommended third party packages for particular tasks (e.g. "regex" as an enhanced alternative to "re", "requests" as an enhanced HTTPS-centric alternative to "urllib.request") 2. Add a new `install_recommended` boolean flag to ensurepip.bootstrap 3. Add a corresponding `--install-recommended flag to the `python -m ensurepip` CLI 4. Add a corresponding `--install-recommended flag to the `python -m venv` CLI (when combined with `--without-pip`, this would run pip directly from the bundled wheel file to do the installations) We'd also need either a new informational PEP or else a section in the developer guide to say that the contents of `ensurepip.RECOMMENDED_PACKAGES` are up to the individual module maintainers (hence keying the mapping by standard library module name, rather than having a single flat list for the entire standard library). For redistributors with weak dependency support, these reference interpreter level recommendations could become redistributor level recommendations. Redistributors without weak dependency support could still make a distinction between "default" installations (which would include them) and "minimal" installations (which would exclude them). Folks writing scripts and example code for independent distribution (i.e. no explicitly declared dependencies) could then choose between relying on just the standard library (as now), or on the standard library plus independently versioned recommended packages. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia [View Less]

13 39

Re: [Python-ideas] Add single() to itertools
by Greg Ewing Oct. 31, 2017

Oct. 31, 2017

2 1

Re: [Python-ideas] Python-ideas Digest, Vol 131, Issue 106
by Ivan Pozdeev Oct. 31, 2017

Oct. 31, 2017

On 31.10.2017 8:37, python-ideas-request(a)python.org wrote: > On Tue, Oct 31, 2017 at 3:50 PM, Ivan Pozdeev via Python-ideas > <python-ideas(a)python.org> wrote: >> On 30.10.2017 17:32, Guido van Rossum wrote: >>> This is a key example of a case where code speaks. Can you write an >>> implementation of how you would want single() to work in Python code? >>> >>> On Mon, Oct 30, 2017 at 2:49 AM, Ivan Pozdeev via Python-ideas >>> … [View More]

6 8

Add processor generation to wheel metadata
by Ivan Pozdeev Oct. 31, 2017

Oct. 31, 2017

Generally, packages are compiled for the same processor generation as the corresponding Python. But not always -- e.g. NumPy opted for SSE2 even for Py2 to work around some compiler bug (https://github.com/numpy/numpy/issues/6428). I was bitten by that at an old machine once and found out that there is no way for `pip' to have checked for that. Besides, performance-oriented packages like the one mentioned could probably benefit from newer instructions. Regarding identifiers: gcc, cl and … [View More]

2 1

IMAP4.__exit__ counterintuitive for with blocks
by Drew Oct. 30, 2017

Oct. 30, 2017

IMAP4.close closes the selected inbox and commits changes such as deletions. It is not called on IMAP4.__exit__ (only logout is, which doesn't call close in its call stack) however, so: with imaplib.IMAP4_SSL(...) as i: ... would fail to commit those changes. close must be explicitly invoked i.e. with imaplib.IMAP4_SSL(...) as i: ... i.close() This is counterintuitive however, as the with statement is meant to automatically clean up. … [View More]

3 3

Thread.__init__ should call super()
by Ilya Kulakov Oct. 30, 2017

Oct. 30, 2017

Since one of the legit use-cases of using the Thread class is subclassing, I think it's __init__ should call super() to support cooperative inheritance. Or perhaps there is a good reason for not doing so? Best Regards, Ilya Kulakov

5 9

Add single() to itertools
by Ivan Pozdeev Oct. 30, 2017

Oct. 30, 2017

The eponymous C#'s LINQ method, I found very useful in the following, quite recurring use-case: I need to get a specific element from a data structure that only supports search semantics (i.e. returns a sequence/iterator of results). For that, I specify very precise search criteria, so only that one item is supposed to be found. But I'd rather verify that just in case. "A data structure that only supports search semantics" is a recurring phenomenon due to this: I make a special-purpose … [View More]

3 2

Composition over Inheritance
by Soni L. Oct. 30, 2017

Oct. 30, 2017

As recent threads indicate, composition may sometimes be better than inheritance. And so I'd like to propose composition as a built-in feature. My idea is syntax of the form o.[c].m(), where o is an object, c is a component, m is a method. I am not sure how you'd set components, or test for components, but I don't think it makes sense to be able to do o.[c][x] or x=o.[c], because those break the idea of automatically passing the object as an argument (unless we create whole wrapper … [View More]

9 22

Re: [Python-ideas] Composition over Inheritance
by Koos Zevenhoven Oct. 29, 2017

Oct. 29, 2017

(looks like you forgot to reply to the list) On Sun, Oct 29, 2017 at 7:47 AM, Greg Ewing <greg.ewing(a)canterbury.ac.nz> wrote: > Koos Zevenhoven wrote: > >> It looks like you can just as easily drive the car without having the key >> > > That's been found to be a problem in real life, too. > More than one Python-programming car thief has been > caught with this function in their personal library: > > def hotwire(car): > car.engine.start() … [View More]

1 0

Dollar operator suggestion
by Yan Pas Oct. 29, 2017

Oct. 29, 2017

I've looked up this feature in haskell. Dollar sign operator is used to avoid parentheses. Rationalle: Python tends to use functions instead of methods ( e.g. len([1,2,3]) instead of [1,2,3].len() ). Sometimes the expression inside parentheses may become big and using a lot of parentheses may tend to bad readability. I suggest the following syntax: len $ [1,2,3] Functions map be also chained: len $ list $ map(...) This operator may be used for function composition too: foo = len $ set $ … [View More]

10 9

PEP Post-History
by Barry Warsaw Oct. 27, 2017

Oct. 27, 2017

We’ve made a small change to the PEP process which may affect readers of python-list and python-ideas, so I’d like to inform you of it. This change was made to PEP 1 and PEP 12. PEPs must have a Post-History header which records the dates at which the PEP is posted to mailing lists, in order to keep the general Python community in the loop as a PEP moves to final status. Until now, PEPs in development were supposed to be posted at least to python-dev and optionally to python-list[1]. This … [View More]

1 0

Add time.time_ns(): system clock with nanosecond resolution
by Victor Stinner Oct. 20, 2017

Oct. 20, 2017

Hi, I would like to add new functions to return time as a number of nanosecond (Python int), especially time.time_ns(). It would enhance the time.time() clock resolution. In my experience, it decreases the minimum non-zero delta between two clock by 3 times, new "ns" clock versus current clock: 84 ns (2.8x better) vs 239 ns on Linux, and 318 us (2.8x better) vs 894 us on Windows, measured in Python. The question of this long email is if it's worth it to add more "_ns" time functions than … [View More]just time.time_ns()? I would like to add: * time.time_ns() * time.monotonic_ns() * time.perf_counter_ns() * time.clock_gettime_ns() * time.clock_settime_ns() time(), monotonic() and perf_counter() clocks are the 3 most common clocks and users use them to get the best available clock resolution. clock_gettime/settime() are the generic UNIX API to access these clocks and so should also be enhanced to get nanosecond resolution. == Nanosecond resolution == More and more clocks have a frequency in MHz, up to GHz for the "TSC" CPU clock, and so the clocks resolution is getting closer to 1 nanosecond (or even better than 1 ns for the TSC clock!). The problem is that Python returns time as a floatting point number which is usually a 64-bit binary floatting number (in the IEEE 754 format). This type starts to loose nanoseconds after 104 days. Conversion from nanoseconds (int) to seconds (float) and then back to nanoseconds (int) to check if conversions loose precision: # no precision loss >>> x=2**52+1; int(float(x * 1e-9) * 1e9) - x 0 # precision loss! (1 nanosecond) >>> x=2**53+1; int(float(x * 1e-9) * 1e9) - x -1 >>> print(datetime.timedelta(seconds=2**53 / 1e9)) 104 days, 5:59:59.254741 While a system administrator can be proud to have an uptime longer than 104 days, the problem also exists for the time.time() clock which returns the number of seconds since the UNIX epoch (1970-01-01). This clock started to loose nanoseconds since mid-May 1970 (47 years ago): >>> import datetime >>> print(datetime.datetime(1970, 1, 1) + datetime.timedelta(seconds=2**53 / 1e9)) 1970-04-15 05:59:59.254741 == PEP 410 == Five years ago, I proposed a large and complex change in all Python functions returning time to support nanosecond resolution using the decimal.Decimal type: https://www.python.org/dev/peps/pep-0410/ The PEP was rejected for different reasons: * it wasn't clear if hardware clocks really had a resolution of 1 nanosecond, especially when the clock is read from Python, since reading a clock in Python also takes time... * Guido van Rossum rejected the idea of adding a new optional parameter to change the result type: it's an uncommon programming practice (bad design in Python) * decimal.Decimal is not widely used, it might be surprised to get such type == CPython enhancements of the last 5 years == Since this PEP was rejected: * the os.stat_result got 3 fields for timestamps as nanoseconds (Python int): st_atime_ns, st_ctime_ns, st_mtime_ns * Python 3.3 got 3 new clocks: time.monotonic(), time.perf_counter() and time.process_time() * I enhanced the private C API of Python handling time (API called "pytime") to store all timings as the new _PyTime_t type which is a simple 64-bit signed integer. The unit of _PyTime_t is not part of the API, it's an implementation detail. The unit is currently 1 nanosecond. This week, I converted one of the last clock to new _PyTime_t format: time.perf_counter() now has internally a resolution of 1 nanosecond, instead of using the C double type. XXX technically https://github.com/python/cpython/pull/3983 is not merged yet :-) == Clocks resolution in Python == I implemented time.time_ns(), time.monotonic_ns() and time.perf_counter_ns() which are similar of the functions without the "_ns" suffix, but return time as nanoseconds (Python int). I computed the smallest difference between two clock reads (ignoring a differences of zero): Linux: * time_ns(): 84 ns <=== !!! * time(): 239 ns <=== !!! * perf_counter_ns(): 84 ns * perf_counter(): 82 ns * monotonic_ns(): 84 ns * monotonic(): 81 ns Windows: * time_ns(): 318000 ns <=== !!! * time(): 894070 ns <=== !!! * perf_counter_ns(): 100 ns * perf_counter(): 100 ns * monotonic_ns(): 15000000 ns * monotonic(): 15000000 ns The difference on time.time() is significant: 84 ns (2.8x better) vs 239 ns on Linux and 318 us (2.8x better) vs 894 us on Windows. The difference will be larger next years since every day adds 864,00,000,000,000 nanoseconds to the system clock :-) (please don't bug me with leap seconds! you got my point) The difference on perf_counter and monotonic clocks are not visible in this quick script since my script runs less than 1 minute, my computer uptime is smaller than 1 weak, ... and Python internally starts these clocks at zero *to reduce the precision loss*! Using an uptime larger than 104 days, you would probably see a significant difference (at least +/- 1 nanosecond) between the regular (seconds as double) and the "_ns" (nanoseconds as int) clocks. == How many new nanosecond clocks? == The PEP 410 proposed to modify the following functions: * os module: fstat(), fstatat(), lstat(), stat() (st_atime, st_ctime and st_mtime fields of the stat structure), sched_rr_get_interval(), times(), wait3() and wait4() * resource module: ru_utime and ru_stime fields of getrusage() * signal module: getitimer(), setitimer() * time module: clock(), clock_gettime(), clock_getres(), monotonic(), time() and wallclock() ("wallclock()" was finally called "monotonic", see PEP 418) According to my tests of the previous section, the precision loss starts after 104 days (stored in nanoseconds). I don't know if it's worth it to modify functions which return "CPU time" or "process time" of processes, since most processes live shorter than 104 days. Do you care of a resolution of 1 nanosecond for the CPU and process time? Maybe we need 1 nanosecond resolution for profiling and benchmarks. But in that case, you might want to implement your profiler in C rather in Python, like the hotshot module, no? The "pytime" private API of CPython gives you clocks with a resolution of 1 nanosecond. == Annex: clock performance == To have an idea of the cost of reading the clock on the clock resolution in Python, I also ran a microbenchmark on *reading* a clock. Example: $ ./python -m perf timeit --duplicate 1024 -s 'import time; t=time.time' 't()' Linux (Mean +- std dev): * time.time(): 45.4 ns +- 0.5 ns * time.time_ns(): 47.8 ns +- 0.8 ns * time.perf_counter(): 46.8 ns +- 0.7 ns * time.perf_counter_ns(): 46.0 ns +- 0.6 ns Windows (Mean +- std dev): * time.time(): 42.2 ns +- 0.8 ns * time.time_ns(): 49.3 ns +- 0.8 ns * time.perf_counter(): 136 ns +- 2 ns <=== * time.perf_counter_ns(): 143 ns +- 4 ns <=== * time.monotonic(): 38.3 ns +- 0.9 ns * time.monotonic_ns(): 48.8 ns +- 1.2 ns Most clocks have the same performance except of perf_counter on Windows: around 140 ns whereas other clocks are around 45 ns (on Linux and Windows): 3x slower. Maybe the "bad" perf_counter performance can be explained by the fact that I'm running Windows in a VM, which is not ideal for benchmarking. Or maybe my C implementation of time.perf_counter() is slow? Note: I expect that a significant part of the numbers are the cost of Python function calls. Reading these clocks using the Python C functions are likely faster. Victor [View Less]

19 53

PEP draft: context variables
by Koos Zevenhoven Oct. 18, 2017

Oct. 18, 2017

Hi all, as promised, here is a draft PEP for context variable semantics and implementation. Apologies for the slight delay; I had a not-so-minor autosave accident and had to retype the majority of this first draft. During the past years, there has been growing interest in something like task-local storage or async-local storage. This PEP proposes an alternative approach to solving the problems that are typically stated as motivation for such concepts. This proposal is based on sketches of … [View More]solutions since spring 2015, with some minor influences from the recent discussion related to PEP 550. I can also see some potential implementation synergy between this PEP and PEP 550, even if the proposed semantics are quite different. So, here it is. This is the first draft and some things are still missing, but the essential things should be there. -- Koos |||||||||||||||||||||||||||||||||||||||||||||||||||||||||| PEP: 999 Title: Context-local variables (contextvars) Version: $Revision$ Last-Modified: $Date$ Author: Koos Zevenhoven Status: Draft Type: Standards Track Content-Type: text/x-rst Created: DD-Mmm-YYYY Post-History: DD-Mmm-YYYY Abstract ======== Sometimes, in special cases, it is desired that code can pass information down the function call chain to the callees without having to explicitly pass the information as arguments to each function in the call chain. This proposal describes a construct which allows code to explicitly switch in and out of a context where a certain context variable has a given value assigned to it. This is a modern alternative to some uses of things like global variables in traditional single-threaded (or thread-unsafe) code and of thread-local storage in traditional *concurrency-unsafe* code (single- or multi-threaded). In particular, the proposed mechanism can also be used with more modern concurrent execution mechanisms such as asynchronously executed coroutines, without the concurrently executed call chains interfering with each other's contexts. The "call chain" can consist of normal functions, awaited coroutines, or generators. The semantics of context variable scope are equivalent in all cases, allowing code to be refactored freely into *subroutines* (which here refers to functions, sub-generators or sub-coroutines) without affecting the semantics of context variables. Regarding implementation, this proposal aims at simplicity and minimum changes to the CPython interpreter and to other Python interpreters. Rationale ========= Consider a modern Python *call chain* (or call tree), which in this proposal refers to any chained (nested) execution of *subroutines*, using any possible combinations of normal function calls, or expressions using ``await`` or ``yield from``. In some cases, passing necessary *information* down the call chain as arguments can substantially complicate the required function signatures, or it can even be impossible to achieve in practice. In these cases, one may search for another place to store this information. Let us look at some historical examples. The most naive option is to assign the value to a global variable or similar, where the code down the call chain can access it. However, this immediately makes the code thread-unsafe, because with multiple threads, all threads assign to the same global variable, and another thread can interfere at any point in the call chain. A somewhat less naive option is to store the information as per-thread information in thread-local storage, where each thread has its own "copy" of the variable which other threads cannot interfere with. Although non-ideal, this has been the best solution in many cases. However, thanks to generators and coroutines, the execution of the call chain can be suspended and resumed, allowing code in other contexts to run concurrently. Therefore, using thread-local storage is *concurrency-unsafe*, because other call chains in other contexts may interfere with the thread-local variable. Note that in the above two historical approaches, the stored information has the *widest* available scope without causing problems. For a third solution along the same path, one would first define an equivalent of a "thread" for asynchronous execution and concurrency. This could be seen as the largest amount of code and nested calls that is guaranteed to be executed sequentially without ambiguity in execution order. This might be referred to as concurrency-local or task-local storage. In this meaning of "task", there is no ambiguity in the order of execution of the code within one task. (This concept of a task is close to equivalent to a ``Task`` in ``asyncio``, but not exactly.) In such concurrency-locals, it is possible to pass information down the call chain to callees without another code path interfering with the value in the background. Common to the above approaches is that they indeed use variables with a wide but just-narrow-enough scope. Thread-locals could also be called thread-wide globals---in single-threaded code, they are indeed truly global. And task-locals could be called task-wide globals, because tasks can be very big. The issue here is that neither global variables, thread-locals nor task-locals are really meant to be used for this purpose of passing information of the execution context down the call chain. Instead of the widest possible variable scope, the scope of the variables should be controlled by the programmer, typically of a library, to have the desired scope---not wider. In other words, task-local variables (and globals and thread-locals) have nothing to do with the kind of context-bound information passing that this proposal intends to enable, even if task-locals can be used to emulate the desired semantics. Therefore, in the following, this proposal describes the semantics and the outlines of an implementation for *context-local variables* (or context variables, contextvars). In fact, as a side effect of this PEP, an async framework can use the proposed feature to implement task-local variables. Proposal ======== Because the proposed semantics are not a direct extension to anything already available in Python, this proposal is first described in terms of semantics and API at a fairly high level. In particular, Python ``with`` statements are heavily used in the description, as they are a good match with the proposed semantics. However, the underlying ``__enter__`` and ``__exit__`` methods correspond to functions in the lower-level speed-optimized (C) API. For clarity of this document, the lower-level functions are not explicitly named in the definition of the semantics. After describing the semantics and high-level API, the implementation is described, going to a lower level. Semantics and higher-level API ------------------------------ Core concept '''''''''''' A context-local variable is represented by a single instance of ``contextvars.Var``, say ``cvar``. Any code that has access to the ``cvar`` object can ask for its value with respect to the current context. In the high-level API, this value is given by the ``cvar.value`` property:: cvar = contextvars.Var(default="the default value", description="example context variable") assert cvar.value == "the default value" # default still applies # In code examples, all ``assert`` statements should # succeed according to the proposed semantics. No assignments to ``cvar`` have been applied for this context, so ``cvar.value`` gives the default value. Assigning new values to contextvars is done in a highly scope-aware manner:: with cvar.assign(new_value): assert cvar.value is new_value # Any code here, or down the call chain from here, sees: # cvar.value is new_value # unless another value has been assigned in a # nested context assert cvar.value is new_value # the assignment of ``cvar`` to ``new_value`` is no longer visible assert cvar.value == "the default value" Here, ``cvar.assign(value)`` returns another object, namely ``contextvars.Assignment(cvar, new_value)``. The essential part here is that applying a context variable assignment (``Assignment.__enter__``) is paired with a de-assignment (``Assignment.__exit__``). These operations set the bounds for the scope of the assigned value. Assignments to the same context variable can be nested to override the outer assignment in a narrower context:: assert cvar.value == "the default value" with cvar.assign("outer"): assert cvar.value == "outer" with cvar.assign("inner"): assert cvar.value == "inner" assert cvar.value == "outer" assert cvar.value == "the default value" Also multiple variables can be assigned to in a nested manner without affecting each other:: cvar1 = contextvars.Var() cvar2 = contextvars.Var() assert cvar1.value is None # default is None by default assert cvar2.value is None with cvar1.assign(value1): assert cvar1.value is value1 assert cvar2.value is None with cvar2.assign(value2): assert cvar1.value is value1 assert cvar2.value is value2 assert cvar1.value is value1 assert cvar2.value is None assert cvar1.value is None assert cvar2.value is None Or with more convenient Python syntax:: with cvar1.assign(value1), cvar2.assign(value2): assert cvar1.value is value1 assert cvar2.value is value2 In another *context*, in another thread or otherwise concurrently executed task or code path, the context variables can have a completely different state. The programmer thus only needs to worry about the context at hand. Refactoring into subroutines '''''''''''''''''''''''''''' Code using contextvars can be refactored into subroutines without affecting the semantics. For instance:: assi = cvar.assign(new_value) def apply(): assi.__enter__() assert cvar.value == "the default value" apply() assert cvar.value is new_value assi.__exit__() assert cvar.value == "the default value" Or similarly in an asynchronous context where ``await`` expressions are used. The subroutine can now be a coroutine:: assi = cvar.assign(new_value) async def apply(): assi.__enter__() assert cvar.value == "the default value" await apply() assert cvar.value is new_value assi.__exit__() assert cvar.value == "the default value" Or when the subroutine is a generator:: def apply(): yield assi.__enter__() which is called using ``yield from apply()`` or with calls to ``next`` or ``.send``. This is discussed further in later sections. Semantics for generators and generator-based coroutines ''''''''''''''''''''''''''''''''''''''''''''''''''''''' Generators, coroutines and async generators act as subroutines in much the same way that normal functions do. However, they have the additional possibility of being suspended by ``yield`` expressions. Assignment contexts entered inside a generator are normally preserved across yields:: def genfunc(): with cvar.assign(new_value): assert cvar.value is new_value yield assert cvar.value is new_value g = genfunc() next(g) assert cvar.value == "the default value" with cvar.assign(another_value): next(g) However, the outer context visible to the generator may change state across yields:: def genfunc(): assert cvar.value is value2 yield assert cvar.value is value1 yield with cvar.assign(value3): assert cvar.value is value3 with cvar.assign(value1): g = genfunc() with cvar.assign(value2): next(g) next(g) next(g) assert cvar.value is value1 Similar semantics apply to async generators defined by ``async def ... yield ...`` ). By default, values assigned inside a generator do not leak through yields to the code that drives the generator. However, the assignment contexts entered and left open inside the generator *do* become visible outside the generator after the generator has finished with a ``StopIteration`` or another exception:: assi = cvar.assign(new_value) def genfunc(): yield assi.__enter__(): yield g = genfunc() assert cvar.value == "the default value" next(g) assert cvar.value == "the default value" next(g) # assi.__enter__() is called here assert cvar.value == "the default value" next(g) assert cvar.value is new_value assi.__exit__() Special functionality for framework authors ------------------------------------------- Frameworks, such as ``asyncio`` or third-party libraries, can use additional functionality in ``contextvars`` to achieve the desired semantics in cases which are not determined by the Python interpreter. Some of the semantics described in this section are also afterwards used to describe the internal implementation. Leaking yields '''''''''''''' Using the ``contextvars.leaking_yields`` decorator, one can choose to leak the context through ``yield`` expressions into the outer context that drives the generator:: @contextvars.leaking_yields def genfunc(): assert cvar.value == "outer" with cvar.assign("inner"): yield assert cvar.value == "inner" assert cvar.value == "outer" g = genfunc(): with cvar.assign("outer"): assert cvar.value == "outer" next(g) assert cvar.value == "inner" next(g) assert cvar.value == "outer" Capturing contextvar assignments '''''''''''''''''''''''''''''''' Using ``contextvars.capture()``, one can capture the assignment contexts that are entered by a block of code. The changes applied by the block of code can then be reverted and subsequently reapplied, even in another context:: assert cvar1.value is None # default assert cvar2.value is None # default assi1 = cvar1.assign(value1) assi2 = cvar1.assign(value2) with contextvars.capture() as delta: assi1.__enter__() with cvar2.assign("not captured"): assert cvar2.value is "not captured" assi2.__enter__() assert cvar1.value is value2 delta.revert() assert cvar1.value is None assert cvar2.value is None ... with cvar1.assign(1), cvar2.assign(2): delta.reapply() assert cvar1.value is value2 assert cvar2.value == 2 However, reapplying the "delta" if its net contents include deassignments may not be possible (see also Implementation and Open Issues). Getting a snapshot of context state ''''''''''''''''''''''''''''''''''' The function ``contextvars.get_local_state()`` returns an object representing the applied assignments to all context-local variables in the context where the function is called. This can be seen as equivalent to using ``contextvars.capture()`` to capture all context changes from the beginning of execution. The returned object supports methods ``.revert()`` and ``reapply()`` as above. Running code in a clean state ''''''''''''''''''''''''''''' Although it is possible to revert all applied context changes using the above primitives, a more convenient way to run a block of code in a clean context is provided:: with context_vars.clean_context(): # here, all context vars start off with their default values # here, the state is back to what it was before the with block. Implementation -------------- This section describes to a variable level of detail how the described semantics can be implemented. At present, an implementation aimed at simplicity but sufficient features is described. More details will be added later. Alternatively, a somewhat more complicated implementation offers minor additional features while adding some performance overhead and requiring more code in the implementation. Data structures and implementation of the core concept '''''''''''''''''''''''''''''''''''''''''''''''''''''' Each thread of the Python interpreter keeps its on stack of ``contextvars.Assignment`` objects, each having a pointer to the previous (outer) assignment like in a linked list. The local state (also returned by ``contextvars.get_local_state()``) then consists of a reference to the top of the stack and a pointer/weak reference to the bottom of the stack. This allows efficient stack manipulations. An object produced by ``contextvars.capture()`` is similar, but refers to only a part of the stack with the bottom reference pointing to the top of the stack as it was in the beginning of the capture block. Now, the stack evolves according to the assignment ``__enter__`` and ``__exit__`` methods. For example:: cvar1 = contextvars.Var() cvar2 = contextvars.Var() # stack: [] assert cvar1.value is None assert cvar2.value is None with cvar1.assign("outer"): # stack: [Assignment(cvar1, "outer")] assert cvar1.value == "outer" with cvar1.assign("inner"): # stack: [Assignment(cvar1, "outer"), # Assignment(cvar1, "inner")] assert cvar1.value == "inner" with cvar2.assign("hello"): # stack: [Assignment(cvar1, "outer"), # Assignment(cvar1, "inner"), # Assignment(cvar2, "hello")] assert cvar2.value == "hello" # stack: [Assignment(cvar1, "outer"), # Assignment(cvar1, "inner")] assert cvar1.value == "inner" assert cvar2.value is None # stack: [Assignment(cvar1, "outer")] assert cvar1.value == "outer" # stack: [] assert cvar1.value is None assert cvar2.value is None Getting a value from the context using ``cvar1.value`` can be implemented as finding the topmost occurrence of a ``cvar1`` assignment on the stack and returning the value there, or the default value if no assignment is found on the stack. However, this can be optimized to instead be an O(1) operation in most cases. Still, even searching through the stack may be reasonably fast since these stacks are not intended to grow very large. The above description is already sufficient for implementing the core concept. Suspendable frames require some additional attention, as explained in the following. Implementation of generator and coroutine semantics ''''''''''''''''''''''''''''''''''''''''''''''''''' Within generators, coroutines and async generators, assignments and deassignments are handled in exactly the same way as anywhere else. However, some changes are needed in the builtin generator methods ``send``, ``__next__``, ``throw`` and ``close``. Here is the Python equivalent of the changes needed in ``send`` for a generator (here ``_old_send`` refers to the behavior in Python 3.6):: def send(self, value): # if decorated with contextvars.leaking_yields if self.gi_contextvars is LEAK: # nothing needs to be done to leak context through yields :) return self._old_send(value) try: with contextvars.capture() as delta: if self.gi_contextvars: # non-zero captured content from previous iteration self.gi_contextvars.reapply() ret = self._old_send(value) except Exception: raise else: # suspending, revert context changes but delta.revert() self.gi_contextvars = delta return ret The corresponding modifications to the other methods is essentially identical. The same applies to coroutines and async generators. For code that does not use ``contextvars``, the additions are O(1) and essentially reduce to a couple of pointer comparisons. For code that does use ``contextvars``, the additions are still O(1) in most cases. More on implementation '''''''''''''''''''''' The rest of the functionality, including ``contextvars.leaking_yields``, contextvars.capture()``, ``contextvars.get_local_state()`` and ``contextvars.clean_context()`` are in fact quite straightforward to implement, but their implementation will be discussed further in later versions of this proposal. Caching of assigned values is somewhat more complicated, and will be discussed later, but it seems that most cases should achieve O(1) complexity. Backwards compatibility ======================= There are no *direct* backwards-compatibility concerns, since a completely new feature is proposed. However, various traditional uses of thread-local storage may need a smooth transition to ``contextvars`` so they can be concurrency-safe. There are several approaches to this, including emulating task-local storage with a little bit of help from async frameworks. A fully general implementation cannot be provided, because the desired semantics may depend on the design of the framework. Another way to deal with the transition is for code to first look for a context created using ``contextvars``. If that fails because a new-style context has not been set or because the code runs on an older Python version, a fallback to thread-local storage is used. Open Issues =========== Out-of-order de-assignments --------------------------- In this proposal, all variable deassignments are made in the opposite order compared to the preceding assignments. This has two useful properties: it encourages using ``with`` statements to define assignment scope and has a tendency to catch errors early (forgetting a ``.__exit__()`` call often results in a meaningful error. To have this as a requirement requirement is beneficial also in terms of implementation simplicity and performance. Nevertheless, allowing out-of-order context exits is not completely out of the question, and reasonable implementation strategies for that do exist. Rejected Ideas ============== Dynamic scoping linked to subroutine scopes ------------------------------------------- The scope of value visibility should not be determined by the way the code is refactored into subroutines. It is necessary to have per-variable control of the assignment scope. Acknowledgements ================ To be added. References ========== To be added. -- + Koos Zevenhoven + http://twitter.com/k7hoven + [View Less]

19 92

(PEP 555 subtopic) Propagation of context in async code
by Koos Zevenhoven Oct. 15, 2017

Oct. 15, 2017

This is a continuation of the PEP 555 discussion in https://mail.python.org/pipermail/python-ideas/2017-September/046916.html And this month in https://mail.python.org/pipermail/python-ideas/2017-October/047279.html If you are new to the discussion, the best point to start reading this might be at my second full paragraph below ("The status quo..."). On Fri, Oct 13, 2017 at 10:25 AM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > On 13 October 2017 at 10:56, Guido van Rossum <… [View More]guido(a)python.org> wrote: > >> I'm out of energy to debate every point (Steve said it well -- that >> decimal/generator example is too contrived), but I found one nit in Nick's >> email that I wish to correct. >> >> On Wed, Oct 11, 2017 at 1:28 AM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: >>> >>> As a less-contrived example, consider context managers implemented as >>> generators. >>> >>> We want those to run with the execution context that's active when >>> they're used in a with statement, not the one that's active when they're >>> created (the fact that generator-based context managers can only be used >>> once mitigates the risk of creation time context capture causing problems, >>> but the implications would still be weird enough to be worth avoiding). >>> >> >> Here I think we're in agreement about the desired semantics, but IMO all >> this requires is some special casing for @contextlib.contextmanager. To me >> this is the exception, not the rule -- in most *other* places I would want >> the yield to switch away from the caller's context. >> >> >>> For native coroutines, we want them to run with the execution context >>> that's active when they're awaited or when they're prepared for submission >>> to an event loop, not the one that's active when they're created. >>> >> >> This caught my eye as wrong. Considering that asyncio's tasks (as well as >> curio's and trio's) *are* native coroutines, we want complete isolation >> between the context active when `await` is called and the context active >> inside the `async def` function. >> > > The rationale for this behaviour *does* arise from a refactoring argument: > > async def original_async_function(): > with some_context(): > do_some_setup() > raw_data = await some_operation() > data = do_some_postprocessing(raw_data) > > Refactored: > > async def async_helper_function(): > do_some_setup() > raw_data = await some_operation() > return do_some_postprocessing(raw_data) > > async def refactored_async_function(): > with some_context(): > data = await async_helper_function() > > *This* type of refactoring argument I *do* subscribe to. > However, considering that coroutines are almost always instantiated at the > point where they're awaited, I do concede that creation time context > capture would likely also work out OK for the coroutine case, which would > leave contextlib.contextmanager as the only special case (and it would turn > off both creation-time context capture *and* context isolation). > The difference between context propagation through coroutine function calls and awaits comes up when you need help from "the" event loop, which means things like creating new tasks from coroutines. However, we cannot even assume that the loop is the only one. So far, it makes no difference where you call the coroutine function. It is only when you await it or schedule it for execution in a loop when something can actually happen. The status quo is that there's nothing that prevents you from calling a coroutine function from within one event loop and then awaiting it in another. So if we want an event loop to be able to pass information down the call chain in such a way that the information is available *throughout the whole task that it is driving*, then the contexts needs to a least propagate through `await`s. This was my starting point 2.5 years ago, when Yury was drafting this status quo (PEP 492). It looked a lot of PEP 492 was inevitable, but that there will be a problem, where each API that uses "blocking IO" somewhere under the hood would need a duplicate version for asyncio (and one for each third-party async framework!). I felt it was necessary to think about a solution before PEP 492 is accepted, and this became a fairly short-lived thread here on python-ideas: https://mail.python.org/pipermail/python-ideas/2015-May/033267.html This year, the discussion on Yury's PEP 550 somehow ended up with a very similar need before I got involved, apparently for independent reasons. A design for solving this need (and others) is also found in my first draft of PEP 555, found at https://mail.python.org/pipermail/python-ideas/2017-September/046916.html Essentially, it's a way of *passing information down the call chain* when it's inconvenient or impossible to pass the information as normal function arguments. I now call the concept "context arguments". More recently, I put some focus on the direct needs of normal users (as opposed direct needs of async framework authors). Those thoughts are most "easily" discussed in terms of generator functions, which are very similar to coroutine functions: A generator function is often thought of as a function that returns an iterable of lazily evaluated values. In this type of usage, the relevant "function call" happens when calling the generator function. The subsequent calls to next() (or a yield from) are thought of as merely getting the items in the iterable, even if they do actually run code in the generator's frame. The discussion on this is found starting from this email: https://mail.python.org/pipermail/python-ideas/2017-October/047279.html However, also coroutines are evaluated lazily. The question is, when should we consider the "subroutine call" to happen: when the coroutine function is called, or when the resulting object is awaited. Often these two are indeed on the same line of code, so it does not matter. But as I discuss above, there are definitely cases where it matters. This has mostly to do with the interactions of different tasks within one event loop, or code where multiple event loops interact. As mentioned above, there are cases where propagating the context through next() and await is desirable. However, there are also cases where the coroutine call is important. This comes up in the case of multiple interacting tasks or multiple event loops. To start with, probably a more example-friendly case, however, is running an event loop and a coroutine from synchronous code: import asyncio async def do_something_context_aware(): do_something_that_depends_on(current_context()) loop = asyncio.get_event_loop() with some_context(): coro = do_something_context_aware() loop.run_until_complete(coro) Now, if the coroutine function call `do_something_context_aware()` does not save the current context on `coro`, then there is no way some_context() can affect the code that will run inside the coroutine, even if that is what we are explicitly trying to do here. The easy solution is to delegate the context transfer to the scheduling function (run_until_complete), and require that the context is passed to that function: with some_context(): coro = do_something_context_aware() loop.run_until_complete(coro) This gives the async framework (here asyncio) a chance to make sure the context propagates as expected. In general, I'm in favor of giving async frameworks some freedom in how this is implemented. However, to give the framework even more freedom, the coroutine call, do_something_context_aware(), could save the current context branch on `coro`, which run_until_complete can attach to the Task that gets created. The bigger question is, what should happen when a coroutine awaits on another coroutine directly, without giving the framework a change to interfere: async def inner(): do_context_aware_stuff() async def outer(): with first_context(): coro = inner() with second_context(): await coro The big question is: In the above, which context should the coroutine be run in? "The" event loop does not have a chance to interfere, so we cannot delegate the decision. We need both versions: the one that propagates first_context() into the coroutine, and the one that propagates second_context() into it. Or, using my metaphor from the other thread, we need "both the forest and the trees". A solution to this would be to have two types of context arguments: 1. (calling) context arguments and 2. execution context arguments Both of these would have their own stack of (argument, value) assignment pairs, explained in the implementation part of the first PEP 555 draft. While this is a complication, the performance overhead of these is so small, that doubling the overhead should not be a performance concern. The question is, do we want these two types of stacks, or do we want to work around it somehow, for instance using context-local storage, implemented on top of the first kind, to implement something like the second kind. However, that again raises some issues of how to propagate the context-local storage down the ambiguous call chain. ––Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + [View Less]

3 6

Add a module itertools.recipes
by Antoine Rozo Oct. 14, 2017

Oct. 14, 2017

Hello, A very useful part of the itertools module's documentation is the section "Recipes", giving utility functions that use itertools iterators. But when you want to use one of theese functions, you have to copy it in your source code (or use external PyPI modules like iterutils). Can we consider making itertools a package and adding a module itertools.recipes that implements all these utilility functions? Regards. -- Antoine Rozo

4 4