Mailman 3 August 2016 - Python-ideas

Re: [Python-ideas] SI scale factors in Python
by Ken Kundert 27 Aug '16

27 Aug '16

All, This proposal basically has two parts. One part is that Python should naturally support printing real numbers with SI scale factors. Currently there are three formats for printing real number: %f, %d, %g. They all become difficult to read for even moderately large or small numbers. Exponential notation is hard for humans to read. That is why SI scale factors have largely replaced exponential notation everywhere except in programming. Adding another format for printing real numbers in human readable form seems like a modest extension that is long past due in all programming languages. I am asking that Python be the leader here. I am sure other languages will pick it up once it is implemented in Python. The second part is the logical dual to the first: input. People should be able to enter numbers in Python using SI scale factors. This means as real literals, such as 2.4G, but it should also work with casting, float('2.4G'). Once you allow SI scale factors on numbers, the natural tendency is for people to want to add units, which is a good thing because it gives important information about the number. We should allow it because it improves the code by making it more self documenting. Even if the language completely ignores the units, we have still gained by allowing the units to be there, just like we gain when we allow user to add comments to their code even though the compiler ignores them. Some people have suggested that we take the next step and use the units for dimensional analysis, but that is highly problematic because you cannot do dimensional analysis unless everything is specified with the correct units, and that can be a huge burden for the user. So instead, I am suggesting that we provide simple hooks that simply allow access to the units. That way people can build dimensional analysis packages using the units if they felt the need. -Ken

10 15

Re: [Python-ideas] SI scale factors in Python
by Ken Kundert 26 Aug '16

26 Aug '16

> So you can have 1000mm or 0.001km but not 1m? If the scale factor is optional, then numbers like 1m are problematic because the m can represent either milli or meter. This is resolved by requiring the scale factor and defining a unity scale factor. I propose '_'. So 1m represents milli and 1_m represents 1 meter. > If units are retained, what you have is no longer a simple number, but > a value with a unit, and is a quite different beast. (For instance, > addition would have to cope with unit mismatches (probably by throwing > an error), and multiplication would have to combine the units (length > * length = area).) That would be a huge new feature. Indeed. I am not proposing that anything be done with the units other than possibly retain them for later output. Doing dimensional analysis on expressions would be a huge burden both for those implementing the language and for those using them in a program. Just allowing the units to be present, even it not retained, is a big advantage because it can bring a great deal of clarity to the meaning of the number. For example, even if the language does not flag an error when a user writes: vdiff = 1mV - 30uA the person that wrote the line will generally see it as a problem and fix it. In my experience, providing units is the most efficient form of documentation available in numerical programming in the sense that one or two additional characters can often clarify otherwise very confusing code. My feeling is that retaining the units on real literals is of little value if you don't also extend the real variable type to hold units, or to create another variable type that would carry the units. Extending reals does not seem like a good idea, but creating a new type, quantity, seems practical. In this case, the units would be rather ephemeral in that they would not survive any operation. Thus, the result of an operation between a quantity and either a integer, real or quantity would always be a real, meaning that the units are lost. In this way, units are very light-weight and only really serve as documentation (for both programmers and end users). But this idea of retaining the units is the least important aspect of this proposal. The important aspects are: 1. It allows numbers to be entered in a clean form that is easy to type and easy to interpret 2. It allows numbers to be output in a clean form that is easy to interpret. 3. In many cases it allows units to be inserted into the code in a very natural and clean way to improve the clarity of the code. > Question, though: What happens with exa-? Currently, if the parser > sees "1E", it'll expect to see another number, eg 1E+1 == 10.0. Will > this double meaning cause confusion? Oh, I did not see this. Both SPICE and Verilog limit the scale factors to the common ones (T, G, M, k, _, m, u, n, p, f, a). I work in electrical engineering, and in that domain exa never comes up. My suggestion would be to limit ourselves to the common scale factors as most people know them. Using P, E, Z, Y, z, and y often actually works against us as most people are not familiar with them and so cannot interpret them easily. -Ken

5 7

Barriers to decimal literals (redux) (was Re: SI scale factors alone, without units or dimensional analysis)
by Nick Coghlan 26 Aug '16

26 Aug '16

On 26 August 2016 at 23:34, Chris Angelico <rosuav(a)gmail.com> wrote: > Introduce "d" as a prefix meaning 1, and this could be the way of > creating something that people have periodically asked for: Decimal > literals. > > (Though IIRC there were some complexities involving Decimal literals > and decimal.getcontext(), which would have to be resolved before 1m > could represent a Decimal.) The most likely candidate for an actual Decimal literal is fixed precision decimal128 - the context dependence of the variable precision decimals currently in the standard library is fine for a custom class, but would be incredibly unintuitive for something that looked similar to a float literal or an int literal (assuming we could get it to work sensibly in the first place). However, actually implementing that would be rather a lot of work, for an unclear pay-off, as the folks that really need decimals seem happy enough with the variable precision ones (especially following Stefan Krah's creation and contribution of the C accelerated implementation), and the vagaries of binary floating point are going to be a factor in the computing landscape for many years to come even if programming language designers do end up collectively settling on decimal128 as the baseline type for decimal data processing. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Re: [Python-ideas] SI scale factors in Python
by Ken Kundert 25 Aug '16

25 Aug '16

Here is a fairly typical example that illustrates the usefulness of supporting SI scale factors and units in Python. This is simple simulation code that is used to test a current mirror driving an 100kOhm resistor ... Here is some simulation code that uses SI scale factors for delta in [-500nA, 0, 500nA]: input = 2.75uA + delta wait(1us) expected = 100kOhm*(2.75uA + delta) tolerance = 2.2mV fails = check_output(expected, tolerance) print('%s: I(in)=%rA, measured V(out)=%rV, expected V(out)=%rV, diff=%rV.' % ( 'FAIL' if fails else 'pass', input, get_output(), expected, get_output() - expected )) with the output being: pass: I(in)=2.25uA, measured V(out)=226.7mV, expected V(out)=225mV, diff=1.7mV. pass: I(in)=2.75uA, measured V(out)=276.8mV, expected V(out)=275mV, diff=1.8mV. FAIL: I(in)=3.25uA, measured V(out)=327.4mV, expected V(out)=325mV, diff=2.4mV. And the same code in Python today ... for delta in [-5e-7, 0, 5e-7]: input = 2.75e-6 + delta wait(1e-6) expected = 1e5*(2.75e-6 + delta) tolerance = 2.2e-3 fails = check_output(expected, tolerance) print('%s: I(in)=%eA, measured V(out)=%eV, expected V(out)=%eV, diff=%eV.' % ( 'FAIL' if fails else 'pass', input, get_output(), expected, get_output() - expected )) with the output being: pass: I(in)=2.25e-6A, measured V(out)=226.7e-3V, expected V(out)=225e-3V, diff=1.7e-3V. pass: I(in)=2.75e-6A, measured V(out)=276.8e-3V, expected V(out)=275e-3V, diff=1.8e-3V. FAIL: I(in)=3.25e-6A, measured V(out)=327.4e-3V, expected V(out)=325e-3V, diff=2.4e-3V. There are two things to notice. In the first example the numbers are easier to read: for example, 500nA is easier to read the 5e-7. Second, there is information in the units that provides provides useful information. One can easily see that the input signal is a current and that the output is a voltage. Furthermore, anybody can look at this code and do a simple sanity check on the expressions even if they don't understand the system being simulated. They can check the units on the input and output expressions to assure that they are all consistent. -Ken

4 3

Adding optional parameter to shutil.rmtree to not delete root.
by Nick Jacobson 25 Aug '16

25 Aug '16

I've been finding that a common scenario is where I want to remove everything in a directory, but leave the (empty) root directory behind, not removing it. So for example, if I have a directory C:\foo and it contains subdirectory C:\foo\bar and file C:\foo\myfile.txt, and I want to remove the subdirectory (and everything in it) and file, leaving only C:\foo behind. (This is useful e.g. when the root directory has special permissions, so it wouldn't be so simple to remove it and recreate it again.) A number of ways to do this have been offered here: http://stackoverflow.com/questions/185936/delete-folder-contents-in-python But it'd be simpler if there were an optional parameter added to shutil.rmtree, called removeroot. It would default to true so as to not break any backward compatibility. If it's set to false, then it leaves the root directory in place. Thanks, Nick

5 5

SI scale factors in Python
by Ken Kundert 25 Aug '16

25 Aug '16

All, I propose that support for SI scale factors be added to Python. This would be very helpful for any program that heavily uses real numbers, such as those involved with scientific and engineering computation. There would be two primary changes. First, the lexer would be enhanced to take real literals with the following forms: c1 = 1nF (same as: c1 = 1e-9 # F ) c = 299.79M (same as: c = 299.79e6 ) f_hy = 1.4204GHz (same as: f_hy = 1.4204e9 # Hz) Basically a scale factor and units may follow a number, both of which are optional, but if the units are given the scale factor must also be given. Any units given could be kept with the number and would be accessible through an attribute or method call, or if it is felt that the cost of storing the units are too high, it may simply be discarded, in which case it is simply serving as documentation. The second change would be to the various real-to-string conversions available in Python. New formatting options would be provided to support SI scale factors. For example, print('Hydrogen line frequency: %q' % f_hy) Hydrogen line frequency: 1.4204GHz If the units are retained with the numbers, then %q (quantity) could be used to print number with SI scale factors and units, and %r (real) could be used to print the number with SI scale factors but without the units. A small package that fleshes out these ideas is available from https://github.com/KenKundert/engfmt It used to be that SI scale factors were only used by scientists and engineers, but over the last 20 years their popularity has increased and so now they are used everywhere. It is time for our programming languages to catch up. I find it a little shocking that no programming languages offer this feature yet, but engineering applications such as SPICE and Verilog have supported SI scale factors for a very long time. -Ken

7 10

Re: [Python-ideas] PEP 525: Asynchronous Generators
by Nathaniel Smith 25 Aug '16

25 Aug '16

Hi Yury et al, Apologies for not participating in this discussion before -- unfortunately I've been dealing with some boring medical issues and haven't been able to contribute as much as early as I should have... But I do have two concerns, one minor and one not. Minor comment: this is purely an aesthetic issue, but the PEP should probably have some discussion of the choice between "async yield" vs bare "yield" as the syntax. Arguments for "async yield" would be consistency with the other async syntax ("async for" etc.) and potentially reduced confusion about how inside an async def, "await" yields, and "yield" yields, but in totally different senses. Arguments for "yield" would be terseness, I guess. Probably this is just something for Guido to pick... Major issue: I have serious concerns about the whole async finalizer design. 1) there is a very general problem of how to handle cleanup in async code, which isn't specific to async generators at all -- really any object that holds resources that need async cleanup has exactly the same issue. So the special purpose solution here makes me uncomfortable. 2) our understanding of this problem (resource cleanup in async code) is *really* immature. You can check the async-sig archives for some discussion -- I think it's safe to say that all the "experts" are extremely confused about this problem's shape/scope/best practices. I guess probably there are 10ish people who kind of understand the problem and 0 people who deeply understand it. (I was working on at least writing something up try and make the issue more accessible, but have not managed yet, which I feel bad about :-(.) But given this, trying to freeze one particular strategy into the language on the 3.6 timeline raises a red flag for me. 3) specifically, I don't understand how the proposed solution in the pep can work reliably in the presence of non-prompt garbage collection (e.g. in the presence of cycles on cpython, or on pypy in general). What happens when the async generator object gets collected after the event loop exits? We could require event loops to force a full collection before exiting, but this is a rather uncomfortable coupling! (Do all python implementations even have a way to force a guaranteed-to-collect-ALL-garbage collection? What are the performance implications of async code triggering such full collections at inopportune times?) And even this didn't necessarily solve the problem, since async generators frames could still be "live" from the GC point of view, even if we know that their resources should be released. (E.g. consider an exception thrown from an async gen that propagates out of the event loop, so the event loop exits while the async gen's frame is pinned by the traceback object.) Python in general has been moving towards more explicit/deterministic cleanup through "with" statements, and I think this is great. My feeling is that we should try to continue in this vein, rather than jumping through hoops to let people get away with sloppy coding that relies on __del__ and usually works except when it doesn't. For example, what if we added an optional __abreak__ coroutine to the async iterator protocol, with the semantics that an "async for" will always either iterate to exhaustion, or else call __abreak__, i.e. # new style code async for x in aiter: ... becomes sugar for # 3.5 style code try: async for x in aiter: ... finally: if for_loop_exited_early and hasattr(aiter, "___abreak__"): await aiter.__abreak__(...) Basically the idea here is that currently, what we should *really* be doing if we care about reliable+deterministic resource cleanup is never writing bare "for" loops but instead always writing async with ... as aiter: async for ... in aiter: ... But that's really tiresome and no one does it, so let's build the "with" into the for loop. Of course there are cases where you want to reuse a single iterator object in multiple loops; this would look like: # doing something clever, so we have to explicitly take charge of cleanup ourselves: async with ... as aiter: # read header (= first non-comment line) async for header in iterlib.unclosing(aiter): if not header.startswith("#"): break # continue with same iterator to read body async for line in iterlib.unclosing(aiter): ... where iterlib.unclosing returns a simple proxy object that passes through __anext__ and friends but swallows __abreak__ to make it a no-op. Or maybe not, this isn't a fully worked out proposal :-). My main concern is that we don't rush to make a decision for 3.6, skip the hard work of considering different designs, and end up with something we regret in retrospect. Some other options to consider: - add async generators to 3.6, but defer the full cleanup discussion for now -- so in 3.6 people will have to use "async with aclosing(aiter): ..." or whatever, but at least we can start getting experience with these objects now, and worry about making them more pleasant to use in the 3.7 time frame. - defer the whole thing to 3.7. Obviously this would be really sad, but... To hold us over until then, it turns out that a library based solution is actually pretty ergonomic -- see https://github.com/njsmith/async_generator . I should add asend/athrow/aclose (I think I have them on a branch), but otherwise it's pretty much as capable and easy to use as "real" async generators. So while I'd love to see these in the language, it doesn't feel super urgent to me. -n On Aug 2, 2016 3:31 PM, "Yury Selivanov" <yselivanov.ml(a)gmail.com> wrote: Hi, This is a new PEP to add asynchronous generators to Python 3.6. The PEP is also available at [1]. There is a reference implementation [2] that supports everything that the PEP proposes to add. [1] https://www.python.org/dev/peps/pep-0525/ [2] https://github.com/1st1/cpython/tree/async_gen Thank you! PEP: 525 Title: Asynchronous Generators Version: $Revision$ Last-Modified: $Date$ Author: Yury Selivanov <yury(a)magic.io> Discussions-To: <python-dev(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Jul-2016 Python-Version: 3.6 Post-History: 02-Aug-2016 Abstract ======== PEP 492 introduced support for native coroutines and ``async``/``await`` syntax to Python 3.5. It is proposed here to extend Python's asynchronous capabilities by adding support for *asynchronous generators*. Rationale and Goals =================== Regular generators (introduced in PEP 255) enabled an elegant way of writing complex *data producers* and have them behave like an iterator. However, currently there is no equivalent concept for the *asynchronous iteration protocol* (``async for``). This makes writing asynchronous data producers unnecessarily complex, as one must define a class that implements ``__aiter__`` and ``__anext__`` to be able to use it in an ``async for`` statement. Essentially, the goals and rationale for PEP 255, applied to the asynchronous execution case, hold true for this proposal as well. Performance is an additional point for this proposal: in our testing of the reference implementation, asynchronous generators are **2x** faster than an equivalent implemented as an asynchronous iterator. As an illustration of the code quality improvement, consider the following class that prints numbers with a given delay once iterated:: class Ticker: """Yield numbers from 0 to `to` every `delay` seconds.""" def __init__(self, delay, to): self.delay = delay self.i = 0 self.to = to def __aiter__(self): return self async def __anext__(self): i = self.i if i >= self.to: raise StopAsyncIteration self.i += 1 if i: await asyncio.sleep(self.delay) return i The same can be implemented as a much simpler asynchronous generator:: async def ticker(delay, to): """Yield numbers from 0 to `to` every `delay` seconds.""" for i in range(to): yield i await asyncio.sleep(delay) Specification ============= This proposal introduces the concept of *asynchronous generators* to Python. This specification presumes knowledge of the implementation of generators and coroutines in Python (PEP 342, PEP 380 and PEP 492). Asynchronous Generators ----------------------- A Python *generator* is any function containing one or more ``yield`` expressions:: def func(): # a function return def genfunc(): # a generator function yield We propose to use the same approach to define *asynchronous generators*:: async def coro(): # a coroutine function await smth() async def asyncgen(): # an asynchronous generator function await smth() yield 42 The result of calling an *asynchronous generator function* is an *asynchronous generator object*, which implements the asynchronous iteration protocol defined in PEP 492. It is a ``SyntaxError`` to have a non-empty ``return`` statement in an asynchronous generator. Support for Asynchronous Iteration Protocol ------------------------------------------- The protocol requires two special methods to be implemented: 1. An ``__aiter__`` method returning an *asynchronous iterator*. 2. An ``__anext__`` method returning an *awaitable* object, which uses ``StopIteration`` exception to "yield" values, and ``StopAsyncIteration`` exception to signal the end of the iteration. Asynchronous generators define both of these methods. Let's manually iterate over a simple asynchronous generator:: async def genfunc(): yield 1 yield 2 gen = genfunc() assert gen.__aiter__() is gen assert await gen.__anext__() == 1 assert await gen.__anext__() == 2 await gen.__anext__() # This line will raise StopAsyncIteration. Finalization ------------ PEP 492 requires an event loop or a scheduler to run coroutines. Because asynchronous generators are meant to be used from coroutines, they also require an event loop to run and finalize them. Asynchronous generators can have ``try..finally`` blocks, as well as ``async with``. It is important to provide a guarantee that, even when partially iterated, and then garbage collected, generators can be safely finalized. For example:: async def square_series(con, to): async with con.transaction(): cursor = con.cursor( 'SELECT generate_series(0, $1) AS i', to) async for row in cursor: yield row['i'] ** 2 async for i in square_series(con, 1000): if i == 100: break The above code defines an asynchronous generator that uses ``async with`` to iterate over a database cursor in a transaction. The generator is then iterated over with ``async for``, which interrupts the iteration at some point. The ``square_series()`` generator will then be garbage collected, and without a mechanism to asynchronously close the generator, Python interpreter would not be able to do anything. To solve this problem we propose to do the following: 1. Implement an ``aclose`` method on asynchronous generators returning a special *awaitable*. When awaited it throws a ``GeneratorExit`` into the suspended generator and iterates over it until either a ``GeneratorExit`` or a ``StopAsyncIteration`` occur. This is very similar to what the ``close()`` method does to regular Python generators, except that an event loop is required to execute ``aclose()``. 2. Raise a ``RuntimeError``, when an asynchronous generator executes a ``yield`` expression in its ``finally`` block (using ``await`` is fine, though):: async def gen(): try: yield finally: await asyncio.sleep(1) # Can use 'await'. yield # Cannot use 'yield', # this line will trigger a # RuntimeError. 3. Add two new methods to the ``sys`` module: ``set_asyncgen_finalizer()`` and ``get_asyncgen_finalizer()``. The idea behind ``sys.set_asyncgen_finalizer()`` is to allow event loops to handle generators finalization, so that the end user does not need to care about the finalization problem, and it just works. When an asynchronous generator is iterated for the first time, it stores a reference to the current finalizer. If there is none, a ``RuntimeError`` is raised. This provides a strong guarantee that every asynchronous generator object will always have a finalizer installed by the correct event loop. When an asynchronous generator is about to be garbage collected, it calls its cached finalizer. The assumption is that the finalizer will schedule an ``aclose()`` call with the loop that was active when the iteration started. For instance, here is how asyncio is modified to allow safe finalization of asynchronous generators:: # asyncio/base_events.py class BaseEventLoop: def run_forever(self): ... old_finalizer = sys.get_asyncgen_finalizer() sys.set_asyncgen_finalizer(self._finalize_asyncgen) try: ... finally: sys.set_asyncgen_finalizer(old_finalizer) ... def _finalize_asyncgen(self, gen): self.create_task(gen.aclose()) ``sys.set_asyncgen_finalizer()`` is thread-specific, so several event loops running in parallel threads can use it safely. Asynchronous Generator Object ----------------------------- The object is modeled after the standard Python generator object. Essentially, the behaviour of asynchronous generators is designed to replicate the behaviour of synchronous generators, with the only difference in that the API is asynchronous. The following methods and properties are defined: 1. ``agen.__aiter__()``: Returns ``agen``. 2. ``agen.__anext__()``: Returns an *awaitable*, that performs one asynchronous generator iteration when awaited. 3. ``agen.asend(val)``: Returns an *awaitable*, that pushes the ``val`` object in the ``agen`` generator. When the ``agen`` has not yet been iterated, ``val`` must be ``None``. Example:: async def gen(): await asyncio.sleep(0.1) v = yield 42 print(v) await asyncio.sleep(0.2) g = gen() await g.asend(None) # Will return 42 after sleeping # for 0.1 seconds. await g.asend('hello') # Will print 'hello' and # raise StopAsyncIteration # (after sleeping for 0.2 seconds.) 4. ``agen.athrow(typ, [val, [tb]])``: Returns an *awaitable*, that throws an exception into the ``agen`` generator. Example:: async def gen(): try: await asyncio.sleep(0.1) yield 'hello' except ZeroDivisionError: await asyncio.sleep(0.2) yield 'world' g = gen() v = await g.asend(None) print(v) # Will print 'hello' after # sleeping for 0.1 seconds. v = await g.athrow(ZeroDivisionError) print(v) # Will print 'world' after $ sleeping 0.2 seconds. 5. ``agen.aclose()``: Returns an *awaitable*, that throws a ``GeneratorExit`` exception into the generator. The *awaitable* can either return a yielded value, if ``agen`` handled the exception, or ``agen`` will be closed and the exception will propagate back to the caller. 6. ``agen.__name__`` and ``agen.__qualname__``: readable and writable name and qualified name attributes. 7. ``agen.ag_await``: The object that ``agen`` is currently *awaiting* on, or ``None``. This is similar to the currently available ``gi_yieldfrom`` for generators and ``cr_await`` for coroutines. 8. ``agen.ag_frame``, ``agen.ag_running``, and ``agen.ag_code``: defined in the same way as similar attributes of standard generators. ``StopIteration`` and ``StopAsyncIteration`` are not propagated out of asynchronous generators, and are replaced with a ``RuntimeError``. Implementation Details ---------------------- Asynchronous generator object (``PyAsyncGenObject``) shares the struct layout with ``PyGenObject``. In addition to that, the reference implementation introduces three new objects: 1. ``PyAsyncGenASend``: the awaitable object that implements ``__anext__`` and ``asend()`` methods. 2. ``PyAsyncGenAThrow``: the awaitable object that implements ``athrow()`` and ``aclose()`` methods. 3. ``_PyAsyncGenWrappedValue``: every directly yielded object from an asynchronous generator is implicitly boxed into this structure. This is how the generator implementation can separate objects that are yielded using regular iteration protocol from objects that are yielded using asynchronous iteration protocol. ``PyAsyncGenASend`` and ``PyAsyncGenAThrow`` are awaitables (they have ``__await__`` methods returning ``self``) and are coroutine-like objects (implementing ``__iter__``, ``__next__``, ``send()`` and ``throw()`` methods). Essentially, they control how asynchronous generators are iterated: .. image:: pep-0525-1.png :align: center :width: 80% PyAsyncGenASend and PyAsyncGenAThrow ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``PyAsyncGenASend`` is a coroutine-like object that drives ``__anext__`` and ``asend()`` methods and implements the asynchronous iteration protocol. ``agen.asend(val)`` and ``agen.__anext__()`` return instances of ``PyAsyncGenASend`` (which hold references back to the parent ``agen`` object.) The data flow is defined as follows: 1. When ``PyAsyncGenASend.send(val)`` is called for the first time, ``val`` is pushed to the parent ``agen`` object (using existing facilities of ``PyGenObject``.) Subsequent iterations over the ``PyAsyncGenASend`` objects, push ``None`` to ``agen``. When a ``_PyAsyncGenWrappedValue`` object is yielded, it is unboxed, and a ``StopIteration`` exception is raised with the unwrapped value as an argument. 2. When ``PyAsyncGenASend.throw(*exc)`` is called for the first time, ``*exc`` is throwed into the parent ``agen`` object. Subsequent iterations over the ``PyAsyncGenASend`` objects, push ``None`` to ``agen``. When a ``_PyAsyncGenWrappedValue`` object is yielded, it is unboxed, and a ``StopIteration`` exception is raised with the unwrapped value as an argument. 3. ``return`` statements in asynchronous generators raise ``StopAsyncIteration`` exception, which is propagated through ``PyAsyncGenASend.send()`` and ``PyAsyncGenASend.throw()`` methods. ``PyAsyncGenAThrow`` is very similar to ``PyAsyncGenASend``. The only difference is that ``PyAsyncGenAThrow.send()``, when called first time, throws an exception into the parent ``agen`` object (instead of pushing a value into it.) New Standard Library Functions and Types ---------------------------------------- 1. ``types.AsyncGeneratorType`` -- type of asynchronous generator object. 2. ``sys.set_asyncgen_finalizer()`` and ``sys.get_asyncgen_finalizer()`` methods to set up asynchronous generators finalizers in event loops. 3. ``inspect.isasyncgen()`` and ``inspect.isasyncgenfunction()`` introspection functions. Backwards Compatibility ----------------------- The proposal is fully backwards compatible. In Python 3.5 it is a ``SyntaxError`` to define an ``async def`` function with a ``yield`` expression inside, therefore it's safe to introduce asynchronous generators in 3.6. Performance =========== Regular Generators ------------------ There is no performance degradation for regular generators. The following micro benchmark runs at the same speed on CPython with and without asynchronous generators:: def gen(): i = 0 while i < 100000000: yield i i += 1 list(gen()) Improvements over asynchronous iterators ---------------------------------------- The following micro-benchmark shows that asynchronous generators are about **2.3x faster** than asynchronous iterators implemented in pure Python:: N = 10 ** 7 async def agen(): for i in range(N): yield i class AIter: def __init__(self): self.i = 0 def __aiter__(self): return self async def __anext__(self): i = self.i if i >= N: raise StopAsyncIteration self.i += 1 return i Design Considerations ===================== ``aiter()`` and ``anext()`` builtins ------------------------------------ Originally, PEP 492 defined ``__aiter__`` as a method that should return an *awaitable* object, resulting in an asynchronous iterator. However, in CPython 3.5.2, ``__aiter__`` was redefined to return asynchronous iterators directly. To avoid breaking backwards compatibility, it was decided that Python 3.6 will support both ways: ``__aiter__`` can still return an *awaitable* with a ``DeprecationWarning`` being issued. Because of this dual nature of ``__aiter__`` in Python 3.6, we cannot add a synchronous implementation of ``aiter()`` built-in. Therefore, it is proposed to wait until Python 3.7. Asynchronous list/dict/set comprehensions ----------------------------------------- Syntax for asynchronous comprehensions is unrelated to the asynchronous generators machinery, and should be considered in a separate PEP. Asynchronous ``yield from`` --------------------------- While it is theoretically possible to implement ``yield from`` support for asynchronous generators, it would require a serious redesign of the generators implementation. ``yield from`` is also less critical for asynchronous generators, since there is no need provide a mechanism of implementing another coroutines protocol on top of coroutines. And to compose asynchronous generators a simple ``async for`` loop can be used:: async def g1(): yield 1 yield 2 async def g2(): async for v in g1(): yield v Why the ``asend()`` and ``athrow()`` methods are necessary ---------------------------------------------------------- They make it possible to implement concepts similar to ``contextlib.contextmanager`` using asynchronous generators. For instance, with the proposed design, it is possible to implement the following pattern:: @async_context_manager async def ctx(): await open() try: yield finally: await close() async with ctx(): await ... Another reason is that it is possible to push data and throw exceptions into asynchronous generators using the object returned from ``__anext__`` object, but it is hard to do that correctly. Adding explicit ``asend()`` and ``athrow()`` will pave a safe way to accomplish that. In terms of implementation, ``asend()`` is a slightly more generic version of ``__anext__``, and ``athrow()`` is very similar to ``aclose()``. Therefore having these methods defined for asynchronous generators does not add any extra complexity. Example ======= A working example with the current reference implementation (will print numbers from 0 to 9 with one second delay):: async def ticker(delay, to): for i in range(to): yield i await asyncio.sleep(delay) async def run(): async for i in ticker(1, 10): print(i) import asyncio loop = asyncio.get_event_loop() try: loop.run_until_complete(run()) finally: loop.close() Implementation ============== The complete reference implementation is available at [1]_. References ========== .. [1] https://github.com/1st1/cpython/tree/async_gen Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

1 0

Re: [Python-ideas] SI scale factors in Python
by Ken Kundert 25 Aug '16

25 Aug '16

> Question, though: What happens with exa-? Currently, if the parser > sees "1E", it'll expect to see another number, eg 1E+1 == 10.0. Will > this double meaning cause confusion? Allow me to refine my answer to this question ... Yes, that is definitely problematic. I see two possible solutions. 1. Limit ourselves to the common scale factors: T, G, M, k, _, m, u, n, p, f, a 2. Or accept X in lieu of E. After all, the e is silent anyway. Thus, on input we accept ... 1Y -> 1e+24 1Z -> 1e+21 -> 1X -> 1e+18 <- only difference 1P -> 1e+15 1T -> 1e+12 1G -> 1e+09 1M -> 1e+06 1k -> 1e+03 1_ -> 1e+00 1m -> 1e-03 1u -> 1e-06 1n -> 1e-09 1p -> 1e-12 1f -> 1e-15 1a -> 1e-18 1z -> 1e-21 1y -> 1e-24 But on output we use ... 1Y -> 1e+24 optional 1Z -> 1e+21 optional -> 1E -> 1e+18 optional 1P -> 1e+15 optional 1T -> 1e+12 1G -> 1e+09 1M -> 1e+06 1k -> 1e+03 1_ -> 1e+00 1m -> 1e-03 1u -> 1e-06 1n -> 1e-09 1p -> 1e-12 1f -> 1e-15 1a -> 1e-18 1z -> 1e-21 optional 1y -> 1e-24 optional The optional scale factors are unfamiliar to most people, and if used might result in harder to read numbers. So I propose that '%r' only outputs the common scale factors, and %R outputs all the scale factors. Or we can use '#' in the format string to indicate the 'alternate' form should be used, in this case 'alternate' means that the extended set of scale factors should be used. -Ken

4 3

Re: [Python-ideas] PEP 525: Asynchronous Generators
by Moritz Sichert 24 Aug '16

24 Aug '16

>>> 2. It's extremely unlikely that somebody will design a system that >>> switches coroutine runners *while async/awaiting a coroutine*. >> Yes, I guess so. >> >> >>> But even in this unlikely use case, you can >>> easily stack finalizers following this pattern: >>> >>> old_finalizer = sys.get_asyncgen_finalizer() >>> sys.set_asyncgen_finalizer(my_finalizer) >>> try: >>> # do my thing >>> finally: >>> sys.set_asyncgen_finalizer(old_finalizer) >> That only works for synchronous code, though, because if this is done in a >> coroutine, it might get suspended within the try block and would leak its >> own finalizer into the outer world. > > set_asyncgen_finalizer is designed to be used *only* by coroutine > runners. This is a low-level API that coroutines should never > touch. (At least my experience working with coroutines says so...) First of all, thanks for your work in this PEP! I think it really completes the async Python to a state where most synchronous code can be changed easily to be asynchronous. I'm not entirely sure about get/set_asyncgen_finalizer(), though. I've written a short function that converts an async iterator (that includes async generators) into a normal one: def sync_iter(async_iter, *, loop=None): async_iter = async_iter.__aiter__() is_asyncgen = inspect.isasyncgen(async_iter) if loop is None: loop = asyncio.get_event_loop() if is_asyncgen: def finalizer(gen): loop.run_until_complete(gen.aclose()) old_finalizer = sys.get_asyncgen_finalizer() sys.set_asyncgen_finalizer(finalizer) try: while True: yield loop.run_until_complete(async_iter.__anext__()) except StopAsyncIteration: return finally: if is_asyncgen: sys.set_asyncgen_finalizer(old_finalizer) Now my questions: - Is it correct do check if the async iterator is actually a generator and only in that case do the whole get/set_asyncgen_finalizer() thing? As I understand, the finalizer is not needed for "normal" async iterators, i.e. instances of classes with a __anext__() method. - Would it make sense to call sys.set_asyncgen_finalizer(old_finalizer) after the first call of async_iter.__anext__() instead of only at the end? As I understand the finalizer is set when the generator is started. - Is loop.run_until_complete(gen.aclose()) a sensible finalizer? If so, is there even any other finalizer that would make sense? Maybe creating a task for gen.aclose() instead of waiting for it to be completed?

2 1

discontinue iterable strings
by Alexander Heger 22 Aug '16

22 Aug '16

standard python should discontinue to see strings as iterables of characters - length-1 strings. I see this as one of the biggest design flaws of python. It may have seem genius at the time, but it has passed it usefulness for practical language use. For example, numpy has no issues >>> np.array('abc') array('abc', dtype='<U3') whereas, as all know, >>> list('abc') ['a', 'b', 'c'] Numpy was of course design a lot later, with more experience in practical use (in mind). Maybe a starting point for transition that latter operation also returns ['abc'] in the long run, could be to have an explicit split operator as recommended use, e.g., 'abc'.split() 'abc'.split('') 'abc'.chars() 'abc'.items() the latter two could return an iterator whereas the former two return lists (currently raise exceptions). Similar for bytes, etc.

14 41