It has been a while since I posted a copy of PEP 1 to the mailing
lists and newsgroups. I've recently done some updating of a few
sections, so in the interest of gaining wider community participation
in the Python development process, I'm posting the latest revision of
PEP 1 here. A version of the PEP is always available on-line at
http://www.python.org/peps/pep-0001.html
Enjoy,
-Barry
-------------------- snip snip --------------------
PEP: 1
Title: PEP Purpose and Guidelines
Version: $Revision: 1.36 $
Last-Modified: $Date: 2002/07/29 18:34:59 $
Author: Barry A. Warsaw, Jeremy Hylton
Status: Active
Type: Informational
Created: 13-Jun-2000
Post-History: 21-Mar-2001, 29-Jul-2002
What is a PEP?
PEP stands for Python Enhancement Proposal. A PEP is a design
document providing information to the Python community, or
describing a new feature for Python. The PEP should provide a
concise technical specification of the feature and a rationale for
the feature.
We intend PEPs to be the primary mechanisms for proposing new
features, for collecting community input on an issue, and for
documenting the design decisions that have gone into Python. The
PEP author is responsible for building consensus within the
community and documenting dissenting opinions.
Because the PEPs are maintained as plain text files under CVS
control, their revision history is the historical record of the
feature proposal[1].
Kinds of PEPs
There are two kinds of PEPs. A standards track PEP describes a
new feature or implementation for Python. An informational PEP
describes a Python design issue, or provides general guidelines or
information to the Python community, but does not propose a new
feature. Informational PEPs do not necessarily represent a Python
community consensus or recommendation, so users and implementors
are free to ignore informational PEPs or follow their advice.
PEP Work Flow
The PEP editor, Barry Warsaw <peps(a)python.org>, assigns numbers
for each PEP and changes its status.
The PEP process begins with a new idea for Python. It is highly
recommended that a single PEP contain a single key proposal or new
idea. The more focussed the PEP, the more successfully it tends
to be. The PEP editor reserves the right to reject PEP proposals
if they appear too unfocussed or too broad. If in doubt, split
your PEP into several well-focussed ones.
Each PEP must have a champion -- someone who writes the PEP using
the style and format described below, shepherds the discussions in
the appropriate forums, and attempts to build community consensus
around the idea. The PEP champion (a.k.a. Author) should first
attempt to ascertain whether the idea is PEP-able. Small
enhancements or patches often don't need a PEP and can be injected
into the Python development work flow with a patch submission to
the SourceForge patch manager[2] or feature request tracker[3].
The PEP champion then emails the PEP editor <peps(a)python.org> with
a proposed title and a rough, but fleshed out, draft of the PEP.
This draft must be written in PEP style as described below.
If the PEP editor approves, he will assign the PEP a number, label
it as standards track or informational, give it status 'draft',
and create and check-in the initial draft of the PEP. The PEP
editor will not unreasonably deny a PEP. Reasons for denying PEP
status include duplication of effort, being technically unsound,
not providing proper motivation or addressing backwards
compatibility, or not in keeping with the Python philosophy. The
BDFL (Benevolent Dictator for Life, Guido van Rossum) can be
consulted during the approval phase, and is the final arbitrator
of the draft's PEP-ability.
If a pre-PEP is rejected, the author may elect to take the pre-PEP
to the comp.lang.python newsgroup (a.k.a. python-list(a)python.org
mailing list) to help flesh it out, gain feedback and consensus
from the community at large, and improve the PEP for
re-submission.
The author of the PEP is then responsible for posting the PEP to
the community forums, and marshaling community support for it. As
updates are necessary, the PEP author can check in new versions if
they have CVS commit permissions, or can email new PEP versions to
the PEP editor for committing.
Standards track PEPs consists of two parts, a design document and
a reference implementation. The PEP should be reviewed and
accepted before a reference implementation is begun, unless a
reference implementation will aid people in studying the PEP.
Standards Track PEPs must include an implementation - in the form
of code, patch, or URL to same - before it can be considered
Final.
PEP authors are responsible for collecting community feedback on a
PEP before submitting it for review. A PEP that has not been
discussed on python-list(a)python.org and/or python-dev(a)python.org
will not be accepted. However, wherever possible, long open-ended
discussions on public mailing lists should be avoided. Strategies
to keep the discussions efficient include, setting up a separate
SIG mailing list for the topic, having the PEP author accept
private comments in the early design phases, etc. PEP authors
should use their discretion here.
Once the authors have completed a PEP, they must inform the PEP
editor that it is ready for review. PEPs are reviewed by the BDFL
and his chosen consultants, who may accept or reject a PEP or send
it back to the author(s) for revision.
Once a PEP has been accepted, the reference implementation must be
completed. When the reference implementation is complete and
accepted by the BDFL, the status will be changed to `Final.'
A PEP can also be assigned status `Deferred.' The PEP author or
editor can assign the PEP this status when no progress is being
made on the PEP. Once a PEP is deferred, the PEP editor can
re-assign it to draft status.
A PEP can also be `Rejected'. Perhaps after all is said and done
it was not a good idea. It is still important to have a record of
this fact.
PEPs can also be replaced by a different PEP, rendering the
original obsolete. This is intended for Informational PEPs, where
version 2 of an API can replace version 1.
PEP work flow is as follows:
Draft -> Accepted -> Final -> Replaced
^
+----> Rejected
v
Deferred
Some informational PEPs may also have a status of `Active' if they
are never meant to be completed. E.g. PEP 1.
What belongs in a successful PEP?
Each PEP should have the following parts:
1. Preamble -- RFC822 style headers containing meta-data about the
PEP, including the PEP number, a short descriptive title
(limited to a maximum of 44 characters), the names, and
optionally the contact info for each author, etc.
2. Abstract -- a short (~200 word) description of the technical
issue being addressed.
3. Copyright/public domain -- Each PEP must either be explicitly
labelled as placed in the public domain (see this PEP as an
example) or licensed under the Open Publication License[4].
4. Specification -- The technical specification should describe
the syntax and semantics of any new language feature. The
specification should be detailed enough to allow competing,
interoperable implementations for any of the current Python
platforms (CPython, JPython, Python .NET).
5. Motivation -- The motivation is critical for PEPs that want to
change the Python language. It should clearly explain why the
existing language specification is inadequate to address the
problem that the PEP solves. PEP submissions without
sufficient motivation may be rejected outright.
6. Rationale -- The rationale fleshes out the specification by
describing what motivated the design and why particular design
decisions were made. It should describe alternate designs that
were considered and related work, e.g. how the feature is
supported in other languages.
The rationale should provide evidence of consensus within the
community and discuss important objections or concerns raised
during discussion.
7. Backwards Compatibility -- All PEPs that introduce backwards
incompatibilities must include a section describing these
incompatibilities and their severity. The PEP must explain how
the author proposes to deal with these incompatibilities. PEP
submissions without a sufficient backwards compatibility
treatise may be rejected outright.
8. Reference Implementation -- The reference implementation must
be completed before any PEP is given status 'Final,' but it
need not be completed before the PEP is accepted. It is better
to finish the specification and rationale first and reach
consensus on it before writing code.
The final implementation must include test code and
documentation appropriate for either the Python language
reference or the standard library reference.
PEP Template
PEPs are written in plain ASCII text, and should adhere to a
rigid style. There is a Python script that parses this style and
converts the plain text PEP to HTML for viewing on the web[5].
PEP 9 contains a boilerplate[7] template you can use to get
started writing your PEP.
Each PEP must begin with an RFC822 style header preamble. The
headers must appear in the following order. Headers marked with
`*' are optional and are described below. All other headers are
required.
PEP: <pep number>
Title: <pep title>
Version: <cvs version string>
Last-Modified: <cvs date string>
Author: <list of authors' real names and optionally, email addrs>
* Discussions-To: <email address>
Status: <Draft | Active | Accepted | Deferred | Final | Replaced>
Type: <Informational | Standards Track>
* Requires: <pep numbers>
Created: <date created on, in dd-mmm-yyyy format>
* Python-Version: <version number>
Post-History: <dates of postings to python-list and python-dev>
* Replaces: <pep number>
* Replaced-By: <pep number>
The Author: header lists the names and optionally, the email
addresses of all the authors/owners of the PEP. The format of the
author entry should be
address(a)dom.ain (Random J. User)
if the email address is included, and just
Random J. User
if the address is not given. If there are multiple authors, each
should be on a separate line following RFC 822 continuation line
conventions. Note that personal email addresses in PEPs will be
obscured as a defense against spam harvesters.
Standards track PEPs must have a Python-Version: header which
indicates the version of Python that the feature will be released
with. Informational PEPs do not need a Python-Version: header.
While a PEP is in private discussions (usually during the initial
Draft phase), a Discussions-To: header will indicate the mailing
list or URL where the PEP is being discussed. No Discussions-To:
header is necessary if the PEP is being discussed privately with
the author, or on the python-list or python-dev email mailing
lists. Note that email addresses in the Discussions-To: header
will not be obscured.
Created: records the date that the PEP was assigned a number,
while Post-History: is used to record the dates of when new
versions of the PEP are posted to python-list and/or python-dev.
Both headers should be in dd-mmm-yyyy format, e.g. 14-Aug-2001.
PEPs may have a Requires: header, indicating the PEP numbers that
this PEP depends on.
PEPs may also have a Replaced-By: header indicating that a PEP has
been rendered obsolete by a later document; the value is the
number of the PEP that replaces the current document. The newer
PEP must have a Replaces: header containing the number of the PEP
that it rendered obsolete.
PEP Formatting Requirements
PEP headings must begin in column zero and the initial letter of
each word must be capitalized as in book titles. Acronyms should
be in all capitals. The body of each section must be indented 4
spaces. Code samples inside body sections should be indented a
further 4 spaces, and other indentation can be used as required to
make the text readable. You must use two blank lines between the
last line of a section's body and the next section heading.
You must adhere to the Emacs convention of adding two spaces at
the end of every sentence. You should fill your paragraphs to
column 70, but under no circumstances should your lines extend
past column 79. If your code samples spill over column 79, you
should rewrite them.
Tab characters must never appear in the document at all. A PEP
should include the standard Emacs stanza included by example at
the bottom of this PEP.
A PEP must contain a Copyright section, and it is strongly
recommended to put the PEP in the public domain.
When referencing an external web page in the body of a PEP, you
should include the title of the page in the text, with a
footnote reference to the URL. Do not include the URL in the body
text of the PEP. E.g.
Refer to the Python Language web site [1] for more details.
...
[1] http://www.python.org
When referring to another PEP, include the PEP number in the body
text, such as "PEP 1". The title may optionally appear. Add a
footnote reference that includes the PEP's title and author. It
may optionally include the explicit URL on a separate line, but
only in the References section. Note that the pep2html.py script
will calculate URLs automatically, e.g.:
...
Refer to PEP 1 [7] for more information about PEP style
...
References
[7] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton
http://www.python.org/peps/pep-0001.html
If you decide to provide an explicit URL for a PEP, please use
this as the URL template:
http://www.python.org/peps/pep-xxxx.html
PEP numbers in URLs must be padded with zeros from the left, so as
to be exactly 4 characters wide, however PEP numbers in text are
never padded.
Reporting PEP Bugs, or Submitting PEP Updates
How you report a bug, or submit a PEP update depends on several
factors, such as the maturity of the PEP, the preferences of the
PEP author, and the nature of your comments. For the early draft
stages of the PEP, it's probably best to send your comments and
changes directly to the PEP author. For more mature, or finished
PEPs you may want to submit corrections to the SourceForge bug
manager[6] or better yet, the SourceForge patch manager[2] so that
your changes don't get lost. If the PEP author is a SF developer,
assign the bug/patch to him, otherwise assign it to the PEP
editor.
When in doubt about where to send your changes, please check first
with the PEP author and/or PEP editor.
PEP authors who are also SF committers, can update the PEPs
themselves by using "cvs commit" to commit their changes.
Remember to also push the formatted PEP text out to the web by
doing the following:
% python pep2html.py -i NUM
where NUM is the number of the PEP you want to push out. See
% python pep2html.py --help
for details.
Transferring PEP Ownership
It occasionally becomes necessary to transfer ownership of PEPs to
a new champion. In general, we'd like to retain the original
author as a co-author of the transferred PEP, but that's really up
to the original author. A good reason to transfer ownership is
because the original author no longer has the time or interest in
updating it or following through with the PEP process, or has
fallen off the face of the 'net (i.e. is unreachable or not
responding to email). A bad reason to transfer ownership is
because you don't agree with the direction of the PEP. We try to
build consensus around a PEP, but if that's not possible, you can
always submit a competing PEP.
If you are interested assuming ownership of a PEP, send a message
asking to take over, addressed to both the original author and the
PEP editor <peps(a)python.org>. If the original author doesn't
respond to email in a timely manner, the PEP editor will make a
unilateral decision (it's not like such decisions can be
reversed. :).
References and Footnotes
[1] This historical record is available by the normal CVS commands
for retrieving older revisions. For those without direct access
to the CVS tree, you can browse the current and past PEP revisions
via the SourceForge web site at
http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/python/nondist/peps/?cvsroot=…
[2] http://sourceforge.net/tracker/?group_id=5470&atid=305470
[3] http://sourceforge.net/tracker/?atid=355470&group_id=5470&func=browse
[4] http://www.opencontent.org/openpub/
[5] The script referred to here is pep2html.py, which lives in
the same directory in the CVS tree as the PEPs themselves.
Try "pep2html.py --help" for details.
The URL for viewing PEPs on the web is
http://www.python.org/peps/
[6] http://sourceforge.net/tracker/?group_id=5470&atid=305470
[7] PEP 9, Sample PEP Template
http://www.python.org/peps/pep-0009.html
Copyright
This document has been placed in the public domain.
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
End:
On 15-04-15, Akira Li <4kir4.1i(a)gmail.com> wrote:
> Isaac Schwabacher <ischwabacher(a)wisc.edu> writes:
>
> > On 15-04-15, Akira Li <4kir4.1i(a)gmail.com> wrote:
> >> Isaac Schwabacher <ischwabacher(a)wisc.edu> writes:
> >> > ...
> >> >
> >> > I know that you can do datetime.now(tz), and you can do datetime(2013,
> >> > 11, 3, 1, 30, tzinfo=zoneinfo('America/Chicago')), but not being able
> >> > to add a time zone to an existing naive datetime is painful (and
> >> > strptime doesn't even let you pass in a time zone).
> >>
> >> `.now(tz)` is correct. `datetime(..., tzinfo=tz)`) is wrong: if tz is a
> >> pytz timezone then you may get a wrong tzinfo (LMT), you should use
> >> `tz.localize(naive_dt, is_dst=False|True|None)` instead.
> >
> > The whole point of this thread is to finalize PEP 431, which fixes the
> > problem for which `localize()` and `normalize()` are workarounds. When
> > this is done, `datetime(..., tzinfo=tz)` will be correct.
> >
> > ijs
>
> The input time is ambiguous. Even if we assume PEP 431 is implemented in
> some form, your code is still missing isdst parameter (or the
> analog). PEP 431 won't fix it; it can't resolve the ambiguity by
> itself. Notice is_dst paramter in the `tz.localize()` call (current
> API).
...yeah, I forgot to throw that in there. It was supposed to be there all along. Nothing to see here, move along.
> .now(tz) works even during end-of-DST transitions (current API) when the
> local time is ambiguous.
I know that. That's what I was complaining about-- I was trying to talk about how astimezone() was going to be inadequate even after the PEP was implemented because it couldn't turn naive datetimes into aware ones, and people were giving examples that started with aware datetimes generated by now(tz), which completely went around the point I was trying to make. But it looks like astimezone() is going to grow an is_dst parameter, and everything will be OK.
ijs
Hi folks,
This is just a note to highlight the fact that I tweaked the "Opting
out" section in PEP 476 based on various discussions I've had over the
past few months: https://hg.python.org/peps/rev/dfd96ee9d6a8
The notable changes:
* the example monkeypatching code handles AttributeError when looking
up "ssl._create_unverified_context", in order to accommodate older
versions of Python that don't have PEP 476 implemented
* new paragraph making it clearer that while the intended use case for
the monkeypatching trick is as a workaround to handle environments
where you *know* HTTPS certificate verification won't work properly
(including explicit references to sitecustomize.py and Standard
Operating Environments for Python), there's also a secondary use case
in allowing applications to provide a system administrator controlled
setting to globally disable certificate verification (hence the change
to the example code)
* new paragraph making it explicit that even though we've improved
Python's default behaviour, particularly security sensitive
applications should still provide their own context rather than
relying on the defaults
Regards,
Nick.
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
# Syntactic sugar
"Beautiful is better than ugly, " thus nice syntax is needed.
Current syntax is very mechanical.
Syntactic sugar is needed on top of current PEP.
# internal vs external
@intify
def foo() -> int:
b = "42"
return b # check 1
x = foo() // 2 # check 2
Does the return type apply to implementation (str) or decorated callable (int)?
How can same annotation or a pair of annotations be used to:
* validate return statement type
* validate subsequent use
* look reasonable in the source code
# lambda
Not mentioned in the PEP, omitted for convenience or is there a rationale?
f = lambda x: None if x is None else str(x ** 2)
Current syntax seems to preclude annotation of `x` due to colon.
Current syntax sort of allows lamba return type annotation, but it's
easy to confuse with `f`.
# local variables
Not mentioned in the PEP
Non-trivial code could really use these.
# global variables
Not mentioned in the PEP
Module-level globals are part of API, annotation is welcome.
What is the syntax?
# comprehensions
[3 * x.data for x in foo if "bar" in x.type]
Arguable, perhaps annotation is only needed on `foo` here, but then
how complex comprehensions, e.g. below, the intermediate comprehension
could use an annotation
[xx for y in [...] if ...]
# class attributes
s = socket.socket(...)
s.type, s.family, s.proto # int
s.fileno # callable
If annotations are only available for methods, it will lead to
Java-style explicit getters and setters.
Python language and data model prefers properties instead, thus
annotations are needed on attributes.
# plain data
user1 = dict(id=123, # always int
name="uuu", # always str
...) # other fields possible
smth = [42, "xx", ...]
(why not namedtuple? b/c extensible, mutable)
At least one PHP IDE allows to annotate PDO.
Perhaps it's just bad taste in Python? Or is there a valid use-case?
# personal note
I think it's amazing how much thought has already been put into this
proposal. The foundation is pretty solid (per Guido talk). I am not at
all opposed to software that infers types (like jedi), or reads
user-specified types (like phpstorm and pep 484) and does something
good with that. In fact I'm ambivalent to current proposal, standard
and promise of better tools on one hand; narrow scope, debatable looks
on the other.
-- dima
Hello,
is it possible to somehow tell Python 2.7 to compile a code entered in the
interactive session with the flag PyCF_SOURCE_IS_UTF8 set? I'm considering
adding support for Python 2 in my package (
https://github.com/Drekin/win-unicode-console) and I have run into the fact
that when u"α" is entered in the interactive session, it results in
u"\xce\xb1" rather than u"\u03b1". As this seems to be a highly specialized
question, I'm asking it here.
Regards, Drekin
>From the PEP:
> Why not a __future__ import
>
> __future__ imports are inconvenient and easy to forget to add.
That is a horrible rationale for not using an import. By that logic we
should have everything in built-ins. ;)
> Working example
> ...
The working example only uses async def and await, not async with
nor async for nor __aenter__, etc., etc.
Could you put in a more complete example -- maybe a basic chat room
with both server and client -- that demonstrated more of the new
possibilities?
Having gone through the PEP again, I am still no closer to understanding
what happens here:
data = await reader.read(8192)
What does the flow of control look like at the interpreter level?
--
~Ethan~
Hi python-dev,
New version of the PEP is attached. Summary of updates:
1. Terminology:
- *native coroutine* term is used for "async def" functions.
- *generator-based coroutine* term is used for PEP 380
coroutines used in asyncio.
- *coroutine* is used when *native coroutine* or
*generator based coroutine* can be used in the same
context.
I think that it's not really productive to discuss the
terminology that we use in the PEP. Its only purpose is
to disambiguate concepts used in the PEP. We should discuss
how we will name new 'async def' coroutines in Python
Documentation if the PEP is accepted. Although if you
notice that somewhere in the PEP it is not crystal clear
what "coroutine" means please give me a heads up!
2. Syntax of await expressions is now thoroghly defined
in the PEP. See "Await Expression", "Updated operator
precedence table", and "Examples of "await" expressions"
sections.
I like the current approach. I'm still not convinced
that we should make 'await' the same grammatically as
unary minus.
I don't understand why we should allow parsing things
like 'await -fut'; this should be a SyntaxError.
I'm fine to modify the grammar to allow 'await await fut'
syntax, though. And I'm open to discussion :)
3. CO_NATIVE_COROUTINE flag. This enables us to disable
__iter__ and __next__ on native coroutines while maintaining
full backwards compatibility.
4. asyncio / Migration strategy. Existing code can
be used with PEP 492 as is, everything will work as
expected.
However, since *plain generator*s (not decorated with
@asyncio.coroutine) cannot 'yield from' native coroutines
(async def functions), it might break some code
*while adapting it to the new syntax*.
I'm open to just throw a RuntimeWarning in this case
in 3.5, and raise a TypeError in 3.6.
Please see the "Differences from generators" section of
the PEP.
5. inspect.isawaitable() function. And, all new functions
are now listed in the "New Standard Library Functions"
section.
6. Lot's of small updates and tweaks throughout the PEP.
Thanks,
Yury
PEP: 492
Title: Coroutines with async and await syntax
Version: $Revision$
Last-Modified: $Date$
Author: Yury Selivanov <yselivanov(a)sprymix.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 09-Apr-2015
Python-Version: 3.5
Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015, 29-Apr-2015
Abstract
========
This PEP introduces new syntax for coroutines, asynchronous ``with``
statements and ``for`` loops. The main motivation behind this proposal
is to streamline writing and maintaining asynchronous code, as well as
to simplify previously hard to implement code patterns.
Rationale and Goals
===================
Current Python supports implementing coroutines via generators (PEP
342), further enhanced by the ``yield from`` syntax introduced in PEP
380. This approach has a number of shortcomings:
* it is easy to confuse coroutines with regular generators, since they
share the same syntax; async libraries often attempt to alleviate
this by using decorators (e.g. ``(a)asyncio.coroutine`` [1]_);
* it is not possible to natively define a coroutine which has no
``yield`` or ``yield from`` statements, again requiring the use of
decorators to fix potential refactoring issues;
* support for asynchronous calls is limited to expressions where
``yield`` is allowed syntactically, limiting the usefulness of
syntactic features, such as ``with`` and ``for`` statements.
This proposal makes coroutines a native Python language feature, and
clearly separates them from generators. This removes
generator/coroutine ambiguity, and makes it possible to reliably define
coroutines without reliance on a specific library. This also enables
linters and IDEs to improve static code analysis and refactoring.
Native coroutines and the associated new syntax features make it
possible to define context manager and iteration protocols in
asynchronous terms. As shown later in this proposal, the new ``async
with`` statement lets Python programs perform asynchronous calls when
entering and exiting a runtime context, and the new ``async for``
statement makes it possible to perform asynchronous calls in iterators.
Specification
=============
This proposal introduces new syntax and semantics to enhance coroutine
support in Python.
This specification presumes knowledge of the implementation of
coroutines in Python (PEP 342 and PEP 380). Motivation for the syntax
changes proposed here comes from the asyncio framework (PEP 3156) and
the "Cofunctions" proposal (PEP 3152, now rejected in favor of this
specification).
From this point in this document we use the word *native coroutine* to
refer to functions declared using the new syntax. *generator-based
coroutine* is used where necessary to refer to coroutines that are
based on generator syntax. *coroutine* is used in contexts where both
definitions are applicable.
New Coroutine Declaration Syntax
--------------------------------
The following new syntax is used to declare a *native coroutine*::
async def read_data(db):
pass
Key properties of *native coroutines*:
* ``async def`` functions are always coroutines, even if they do not
contain ``await`` expressions.
* It is a ``SyntaxError`` to have ``yield`` or ``yield from``
expressions in an ``async`` function.
* Internally, two new code object flags were introduced:
- ``CO_COROUTINE`` is used to enable runtime detection of
*coroutines* (and migrating existing code).
- ``CO_NATIVE_COROUTINE`` is used to mark *native coroutines*
(defined with new syntax.)
All coroutines have ``CO_COROUTINE``, ``CO_NATIVE_COROUTINE``, and
``CO_GENERATOR`` flags set.
* Regular generators, when called, return a *generator object*;
similarly, coroutines return a *coroutine object*.
* ``StopIteration`` exceptions are not propagated out of coroutines,
and are replaced with a ``RuntimeError``. For regular generators
such behavior requires a future import (see PEP 479).
* See also `Coroutine objects`_ section.
types.coroutine()
-----------------
A new function ``coroutine(gen)`` is added to the ``types`` module. It
allows interoperability between existing *generator-based coroutines*
in asyncio and *native coroutines* introduced by this PEP.
The function applies ``CO_COROUTINE`` flag to generator-function's code
object, making it return a *coroutine object*.
The function can be used as a decorator, since it modifies generator-
functions in-place and returns them.
Note, that the ``CO_NATIVE_COROUTINE`` flag is not applied by
``types.coroutine()`` to make it possible to separate *native
coroutines* defined with new syntax, from *generator-based coroutines*.
Await Expression
----------------
The following new ``await`` expression is used to obtain a result of
coroutine execution::
async def read_data(db):
data = await db.fetch('SELECT ...')
...
``await``, similarly to ``yield from``, suspends execution of
``read_data`` coroutine until ``db.fetch`` *awaitable* completes and
returns the result data.
It uses the ``yield from`` implementation with an extra step of
validating its argument. ``await`` only accepts an *awaitable*, which
can be one of:
* A *native coroutine object* returned from a *native coroutine*.
* A *generator-based coroutine object* returned from a generator
decorated with ``types.coroutine()``.
* An object with an ``__await__`` method returning an iterator.
Any ``yield from`` chain of calls ends with a ``yield``. This is a
fundamental mechanism of how *Futures* are implemented. Since,
internally, coroutines are a special kind of generators, every
``await`` is suspended by a ``yield`` somewhere down the chain of
``await`` calls (please refer to PEP 3156 for a detailed
explanation.)
To enable this behavior for coroutines, a new magic method called
``__await__`` is added. In asyncio, for instance, to enable *Future*
objects in ``await`` statements, the only change is to add
``__await__ = __iter__`` line to ``asyncio.Future`` class.
Objects with ``__await__`` method are called *Future-like* objects in
the rest of this PEP.
Also, please note that ``__aiter__`` method (see its definition
below) cannot be used for this purpose. It is a different protocol,
and would be like using ``__iter__`` instead of ``__call__`` for
regular callables.
It is a ``TypeError`` if ``__await__`` returns anything but an
iterator.
* Objects defined with CPython C API with a ``tp_await`` function,
returning an iterator (similar to ``__await__`` method).
It is a ``SyntaxError`` to use ``await`` outside of an ``async def``
function (like it is a ``SyntaxError`` to use ``yield`` outside of
``def`` function.)
It is a ``TypeError`` to pass anything other than an *awaitable* object
to an ``await`` expression.
Updated operator precedence table
'''''''''''''''''''''''''''''''''
``await`` keyword is defined as follows::
power ::= await ["**" u_expr]
await ::= ["await"] primary
where "primary" represents the most tightly bound operations of the
language. Its syntax is::
primary ::= atom | attributeref | subscription | slicing | call
See Python Documentation [12]_ and `Grammar Updates`_ section of this
proposal for details.
The key ``await`` difference from ``yield`` and ``yield from``
operators is that *await expressions* do not require parentheses around
them most of the times.
Also, ``yield from`` allows any expression as its argument, including
expressions like ``yield from a() + b()``, that would be parsed as
``yield from (a() + b())``, which is almost always a bug. In general,
the result of any arithmetic operation is not an *awaitable* object.
To avoid this kind of mistakes, it was decided to make ``await``
precedence lower than ``[]``, ``()``, and ``.``, but higher than ``**``
operators.
+------------------------------+-----------------------------------+
| Operator | Description |
+==============================+===================================+
| ``yield`` ``x``, | Yield expression |
| ``yield from`` ``x`` | |
+------------------------------+-----------------------------------+
| ``lambda`` | Lambda expression |
+------------------------------+-----------------------------------+
| ``if`` -- ``else`` | Conditional expression |
+------------------------------+-----------------------------------+
| ``or`` | Boolean OR |
+------------------------------+-----------------------------------+
| ``and`` | Boolean AND |
+------------------------------+-----------------------------------+
| ``not`` ``x`` | Boolean NOT |
+------------------------------+-----------------------------------+
| ``in``, ``not in``, | Comparisons, including membership |
| ``is``, ``is not``, ``<``, | tests and identity tests |
| ``<=``, ``>``, ``>=``, | |
| ``!=``, ``==`` | |
+------------------------------+-----------------------------------+
| ``|`` | Bitwise OR |
+------------------------------+-----------------------------------+
| ``^`` | Bitwise XOR |
+------------------------------+-----------------------------------+
| ``&`` | Bitwise AND |
+------------------------------+-----------------------------------+
| ``<<``, ``>>`` | Shifts |
+------------------------------+-----------------------------------+
| ``+``, ``-`` | Addition and subtraction |
+------------------------------+-----------------------------------+
| ``*``, ``@``, ``/``, ``//``, | Multiplication, matrix |
| ``%`` | multiplication, division, |
| | remainder |
+------------------------------+-----------------------------------+
| ``+x``, ``-x``, ``~x`` | Positive, negative, bitwise NOT |
+------------------------------+-----------------------------------+
| ``**`` | Exponentiation |
+------------------------------+-----------------------------------+
| ``await`` ``x`` | Await expression |
+------------------------------+-----------------------------------+
| ``x[index]``, | Subscription, slicing, |
| ``x[index:index]``, | call, attribute reference |
| ``x(arguments...)``, | |
| ``x.attribute`` | |
+------------------------------+-----------------------------------+
| ``(expressions...)``, | Binding or tuple display, |
| ``[expressions...]``, | list display, |
| ``{key: value...}``, | dictionary display, |
| ``{expressions...}`` | set display |
+------------------------------+-----------------------------------+
Examples of "await" expressions
'''''''''''''''''''''''''''''''
Valid syntax examples:
================================== ==================================
Expression Will be parsed as
================================== ==================================
``if await fut: pass`` ``if (await fut): pass``
``if await fut + 1: pass`` ``if (await fut) + 1: pass``
``pair = await fut, 'spam'`` ``pair = (await fut), 'spam'``
``with await fut, open(): pass`` ``with (await fut), open(): pass``
``await foo()['spam'].baz()()`` ``await ( foo()['spam'].baz()() )``
``return await coro()`` ``return ( await coro() )``
``res = await coro() ** 2`` ``res = (await coro()) ** 2``
``func(a1=await coro(), a2=0)`` ``func(a1=(await coro()), a2=0)``
``await foo() + await bar()`` ``(await foo()) + (await bar())``
``-await foo()`` ``-(await foo())``
================================== ==================================
Invalid syntax examples:
================================== ==================================
Expression Should be written as
================================== ==================================
``await await coro()`` ``await (await coro())``
``await -coro()`` ``await (-coro())``
================================== ==================================
Asynchronous Context Managers and "async with"
----------------------------------------------
An *asynchronous context manager* is a context manager that is able to
suspend execution in its *enter* and *exit* methods.
To make this possible, a new protocol for asynchronous context managers
is proposed. Two new magic methods are added: ``__aenter__`` and
``__aexit__``. Both must return an *awaitable*.
An example of an asynchronous context manager::
class AsyncContextManager:
async def __aenter__(self):
await log('entering context')
async def __aexit__(self, exc_type, exc, tb):
await log('exiting context')
New Syntax
''''''''''
A new statement for asynchronous context managers is proposed::
async with EXPR as VAR:
BLOCK
which is semantically equivalent to::
mgr = (EXPR)
aexit = type(mgr).__aexit__
aenter = type(mgr).__aenter__(mgr)
exc = True
try:
VAR = await aenter
BLOCK
except:
if not await aexit(mgr, *sys.exc_info()):
raise
else:
await aexit(mgr, None, None, None)
As with regular ``with`` statements, it is possible to specify multiple
context managers in a single ``async with`` statement.
It is an error to pass a regular context manager without ``__aenter__``
and ``__aexit__`` methods to ``async with``. It is a ``SyntaxError``
to use ``async with`` outside of an ``async def`` function.
Example
'''''''
With *asynchronous context managers* it is easy to implement proper
database transaction managers for coroutines::
async def commit(session, data):
...
async with session.transaction():
...
await session.update(data)
...
Code that needs locking also looks lighter::
async with lock:
...
instead of::
with (yield from lock):
...
Asynchronous Iterators and "async for"
--------------------------------------
An *asynchronous iterable* is able to call asynchronous code in its
*iter* implementation, and *asynchronous iterator* can call
asynchronous code in its *next* method. To support asynchronous
iteration:
1. An object must implement an ``__aiter__`` method returning an
*awaitable* resulting in an *asynchronous iterator object*.
2. An *asynchronous iterator object* must implement an ``__anext__``
method returning an *awaitable*.
3. To stop iteration ``__anext__`` must raise a ``StopAsyncIteration``
exception.
An example of asynchronous iterable::
class AsyncIterable:
async def __aiter__(self):
return self
async def __anext__(self):
data = await self.fetch_data()
if data:
return data
else:
raise StopAsyncIteration
async def fetch_data(self):
...
New Syntax
''''''''''
A new statement for iterating through asynchronous iterators is
proposed::
async for TARGET in ITER:
BLOCK
else:
BLOCK2
which is semantically equivalent to::
iter = (ITER)
iter = await type(iter).__aiter__(iter)
running = True
while running:
try:
TARGET = await type(iter).__anext__(iter)
except StopAsyncIteration:
running = False
else:
BLOCK
else:
BLOCK2
It is a ``TypeError`` to pass a regular iterable without ``__aiter__``
method to ``async for``. It is a ``SyntaxError`` to use ``async for``
outside of an ``async def`` function.
As for with regular ``for`` statement, ``async for`` has an optional
``else`` clause.
Example 1
'''''''''
With asynchronous iteration protocol it is possible to asynchronously
buffer data during iteration::
async for data in cursor:
...
Where ``cursor`` is an asynchronous iterator that prefetches ``N`` rows
of data from a database after every ``N`` iterations.
The following code illustrates new asynchronous iteration protocol::
class Cursor:
def __init__(self):
self.buffer = collections.deque()
def _prefetch(self):
...
async def __aiter__(self):
return self
async def __anext__(self):
if not self.buffer:
self.buffer = await self._prefetch()
if not self.buffer:
raise StopAsyncIteration
return self.buffer.popleft()
then the ``Cursor`` class can be used as follows::
async for row in Cursor():
print(row)
which would be equivalent to the following code::
i = await Cursor().__aiter__()
while True:
try:
row = await i.__anext__()
except StopAsyncIteration:
break
else:
print(row)
Example 2
'''''''''
The following is a utility class that transforms a regular iterable to
an asynchronous one. While this is not a very useful thing to do, the
code illustrates the relationship between regular and asynchronous
iterators.
::
class AsyncIteratorWrapper:
def __init__(self, obj):
self._it = iter(obj)
async def __aiter__(self):
return self
async def __anext__(self):
try:
value = next(self._it)
except StopIteration:
raise StopAsyncIteration
return value
async for letter in AsyncIteratorWrapper("abc"):
print(letter)
Why StopAsyncIteration?
'''''''''''''''''''''''
Coroutines are still based on generators internally. So, before PEP
479, there was no fundamental difference between
::
def g1():
yield from fut
return 'spam'
and
::
def g2():
yield from fut
raise StopIteration('spam')
And since PEP 479 is accepted and enabled by default for coroutines,
the following example will have its ``StopIteration`` wrapped into a
``RuntimeError``
::
async def a1():
await fut
raise StopIteration('spam')
The only way to tell the outside code that the iteration has ended is
to raise something other than ``StopIteration``. Therefore, a new
built-in exception class ``StopAsyncIteration`` was added.
Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions
raised in coroutines are wrapped in ``RuntimeError``.
Coroutine objects
-----------------
Differences from generators
'''''''''''''''''''''''''''
This section applies only to *native coroutines* with
``CO_NATIVE_COROUTINE`` flag, i.e. defined with the new ``async def``
syntax.
**The behavior of existing *generator-based coroutines* in asyncio
remains unchanged.**
Great effort has been made to make sure that coroutines and
generators are treated as distinct concepts:
1. *Native coroutine objects* do not implement ``__iter__`` and
``__next__`` methods. Therefore, they cannot be iterated over or
passed to ``iter()``, ``list()``, ``tuple()`` and other built-ins.
They also cannot be used in a ``for..in`` loop.
An attempt to use ``__iter__`` or ``__next__`` on a *native
coroutine object* will result in a ``TypeError``.
2. *Plain generators* cannot ``yield from`` *native coroutine objects*:
doing so will result in a ``TypeError``.
3. *generator-based coroutines* (for asyncio code must be decorated
with ``(a)asyncio.coroutine``) can ``yield from`` *native coroutine
objects*.
4. ``inspect.isgenerator()`` and ``inspect.isgeneratorfunction()``
return ``False`` for *native coroutine objects* and *native
coroutine functions*.
Coroutine object methods
''''''''''''''''''''''''
Coroutines are based on generators internally, thus they share the
implementation. Similarly to generator objects, coroutine objects have
``throw()``, ``send()`` and ``close()`` methods. ``StopIteration`` and
``GeneratorExit`` play the same role for coroutine objects (although
PEP 479 is enabled by default for coroutines). See PEP 342, PEP 380,
and Python Documentation [11]_ for details.
``throw()``, ``send()`` methods for coroutine objects are used to push
values and raise errors into *Future-like* objects.
Debugging Features
------------------
A common beginner mistake is forgetting to use ``yield from`` on
coroutines::
@asyncio.coroutine
def useful():
asyncio.sleep(1) # this will do noting without 'yield from'
For debugging this kind of mistakes there is a special debug mode in
asyncio, in which ``@coroutine`` decorator wraps all functions with a
special object with a destructor logging a warning. Whenever a wrapped
generator gets garbage collected, a detailed logging message is
generated with information about where exactly the decorator function
was defined, stack trace of where it was collected, etc. Wrapper
object also provides a convenient ``__repr__`` function with detailed
information about the generator.
The only problem is how to enable these debug capabilities. Since
debug facilities should be a no-op in production mode, ``@coroutine``
decorator makes the decision of whether to wrap or not to wrap based on
an OS environment variable ``PYTHONASYNCIODEBUG``. This way it is
possible to run asyncio programs with asyncio's own functions
instrumented. ``EventLoop.set_debug``, a different debug facility, has
no impact on ``@coroutine`` decorator's behavior.
With this proposal, coroutines is a native, distinct from generators,
concept. New methods ``set_coroutine_wrapper`` and
``get_coroutine_wrapper`` are added to the ``sys`` module, with which
frameworks can provide advanced debugging facilities.
It is also important to make coroutines as fast and efficient as
possible, therefore there are no debug features enabled by default.
Example::
async def debug_me():
await asyncio.sleep(1)
def async_debug_wrap(generator):
return asyncio.CoroWrapper(generator)
sys.set_coroutine_wrapper(async_debug_wrap)
debug_me() # <- this line will likely GC the coroutine object and
# trigger asyncio.CoroWrapper's code.
assert isinstance(debug_me(), asyncio.CoroWrapper)
sys.set_coroutine_wrapper(None) # <- this unsets any
# previously set wrapper
assert not isinstance(debug_me(), asyncio.CoroWrapper)
New Standard Library Functions
------------------------------
* ``types.coroutine(gen)``. See `types.coroutine()`_ section for
details.
* ``inspect.iscoroutine(obj)`` returns ``True`` if ``obj`` is a
*coroutine object*.
* ``inspect.iscoroutinefunction(obj)`` returns ``True`` if ``obj`` is a
*coroutine function*.
* ``inspect.isawaitable(obj)`` returns ``True`` if ``obj`` can be used
in ``await`` expression. See `Await Expression`_ for details.
* ``sys.set_coroutine_wrapper(wraper)`` allows to intercept creation of
*coroutine objects*. ``wraper`` must be a callable that accepts one
argument: a *coroutine object* or ``None``. ``None`` resets the
wrapper. If called twice, the new wrapper replaces the previous one.
See `Debugging Features`_ for more details.
* ``sys.get_coroutine_wrapper()`` returns the current wrapper object.
Returns ``None`` if no wrapper was set. See `Debugging Features`_
for more details.
Glossary
========
:Native coroutine:
A coroutine function is declared with ``async def``. It uses
``await`` and ``return value``; see `New Coroutine Declaration
Syntax`_ for details.
:Native coroutine object:
Returned from a native coroutine function. See `Await Expression`_
for details.
:Generator-based coroutine:
Coroutines based on generator syntax. Most common example are
functions decorated with ``(a)asyncio.coroutine``.
:Generator-based coroutine object:
Returned from a generator-based coroutine function.
:Coroutine:
Either *native coroutine* or *generator-based coroutine*.
:Coroutine object:
Either *native coroutine object* or *generator-based coroutine
object*.
:Future-like object:
An object with an ``__await__`` method, or a C object with
``tp_await`` function, returning an iterator. Can be consumed by
an ``await`` expression in a coroutine. A coroutine waiting for a
Future-like object is suspended until the Future-like object's
``__await__`` completes, and returns the result. See `Await
Expression`_ for details.
:Awaitable:
A *Future-like* object or a *coroutine object*. See `Await
Expression`_ for details.
:Asynchronous context manager:
An asynchronous context manager has ``__aenter__`` and ``__aexit__``
methods and can be used with ``async with``. See `Asynchronous
Context Managers and "async with"`_ for details.
:Asynchronous iterable:
An object with an ``__aiter__`` method, which must return an
*asynchronous iterator* object. Can be used with ``async for``.
See `Asynchronous Iterators and "async for"`_ for details.
:Asynchronous iterator:
An asynchronous iterator has an ``__anext__`` method. See
`Asynchronous Iterators and "async for"`_ for details.
List of functions and methods
=============================
================= =================================== =================
Method Can contain Can't contain
================= =================================== =================
async def func await, return value yield, yield from
async def __a*__ await, return value yield, yield from
def __a*__ return awaitable await
def __await__ yield, yield from, return iterable await
generator yield, yield from, return value await
================= =================================== =================
Where:
* "async def func": native coroutine;
* "async def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``,
``__aexit__`` defined with the ``async`` keyword;
* "def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``,
``__aexit__`` defined without the ``async`` keyword, must return an
*awaitable*;
* "def __await__": ``__await__`` method to implement *Future-like*
objects;
* generator: a "regular" generator, function defined with ``def`` and
which contains a least one ``yield`` or ``yield from`` expression.
Transition Plan
===============
To avoid backwards compatibility issues with ``async`` and ``await``
keywords, it was decided to modify ``tokenizer.c`` in such a way, that
it:
* recognizes ``async def`` name tokens combination (start of a
native coroutine);
* keeps track of regular functions and native coroutines;
* replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with
``AWAIT`` when in the process of yielding tokens for native
coroutines.
This approach allows for seamless combination of new syntax features
(all of them available only in ``async`` functions) with any existing
code.
An example of having "async def" and "async" attribute in one piece of
code::
class Spam:
async = 42
async def ham():
print(getattr(Spam, 'async'))
# The coroutine can be executed and will print '42'
Backwards Compatibility
-----------------------
This proposal preserves 100% backwards compatibility.
asyncio
-------
``asyncio`` module was adapted and tested to work with coroutines and
new statements. Backwards compatibility is 100% preserved, i.e. all
existing code will work as-is.
The required changes are mainly:
1. Modify ``(a)asyncio.coroutine`` decorator to use new
``types.coroutine()`` function.
2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class.
3. Add ``ensure_task()`` as an alias for ``async()`` function.
Deprecate ``async()`` function.
Migration strategy
''''''''''''''''''
Because *plain generators* cannot ``yield from`` *native coroutine
objects* (see `Differences from generators`_ section for more details),
it is advised to make sure that all generator-based coroutines are
decorated with ``(a)asyncio.coroutine`` *before* starting to use the new
syntax.
Grammar Updates
---------------
Grammar changes are also fairly minimal::
decorated: decorators (classdef | funcdef | async_funcdef)
async_funcdef: ASYNC funcdef
compound_stmt: (if_stmt | while_stmt | for_stmt | try_stmt | with_stmt
| funcdef | classdef | decorated | async_stmt)
async_stmt: ASYNC (funcdef | with_stmt | for_stmt)
power: atom_expr ['**' factor]
atom_expr: [AWAIT] atom trailer*
Transition Period Shortcomings
------------------------------
There is just one.
Until ``async`` and ``await`` are not proper keywords, it is not
possible (or at least very hard) to fix ``tokenizer.c`` to recognize
them on the **same line** with ``def`` keyword::
# async and await will always be parsed as variables
async def outer(): # 1
def nested(a=(await fut)):
pass
async def foo(): return (await fut) # 2
Since ``await`` and ``async`` in such cases are parsed as ``NAME``
tokens, a ``SyntaxError`` will be raised.
To workaround these issues, the above examples can be easily rewritten
to a more readable form::
async def outer(): # 1
a_default = await fut
def nested(a=a_default):
pass
async def foo(): # 2
return (await fut)
This limitation will go away as soon as ``async`` and ``await`` are
proper keywords. Or if it's decided to use a future import for this
PEP.
Deprecation Plans
-----------------
``async`` and ``await`` names will be softly deprecated in CPython 3.5
and 3.6. In 3.7 we will transform them to proper keywords. Making
``async`` and ``await`` proper keywords before 3.7 might make it harder
for people to port their code to Python 3.
Design Considerations
=====================
PEP 3152
--------
PEP 3152 by Gregory Ewing proposes a different mechanism for coroutines
(called "cofunctions"). Some key points:
1. A new keyword ``codef`` to declare a *cofunction*. *Cofunction* is
always a generator, even if there is no ``cocall`` expressions
inside it. Maps to ``async def`` in this proposal.
2. A new keyword ``cocall`` to call a *cofunction*. Can only be used
inside a *cofunction*. Maps to ``await`` in this proposal (with
some differences, see below.)
3. It is not possible to call a *cofunction* without a ``cocall``
keyword.
4. ``cocall`` grammatically requires parentheses after it::
atom: cocall | <existing alternatives for atom>
cocall: 'cocall' atom cotrailer* '(' [arglist] ')'
cotrailer: '[' subscriptlist ']' | '.' NAME
5. ``cocall f(*args, **kwds)`` is semantically equivalent to
``yield from f.__cocall__(*args, **kwds)``.
Differences from this proposal:
1. There is no equivalent of ``__cocall__`` in this PEP, which is
called and its result is passed to ``yield from`` in the ``cocall``
expression. ``await`` keyword expects an *awaitable* object,
validates the type, and executes ``yield from`` on it. Although,
``__await__`` method is similar to ``__cocall__``, but is only used
to define *Future-like* objects.
2. ``await`` is defined in almost the same way as ``yield from`` in the
grammar (it is later enforced that ``await`` can only be inside
``async def``). It is possible to simply write ``await future``,
whereas ``cocall`` always requires parentheses.
3. To make asyncio work with PEP 3152 it would be required to modify
``(a)asyncio.coroutine`` decorator to wrap all functions in an object
with a ``__cocall__`` method, or to implement ``__cocall__`` on
generators. To call *cofunctions* from existing generator-based
coroutines it would be required to use ``costart(cofunc, *args,
**kwargs)`` built-in.
4. Since it is impossible to call a *cofunction* without a ``cocall``
keyword, it automatically prevents the common mistake of forgetting
to use ``yield from`` on generator-based coroutines. This proposal
addresses this problem with a different approach, see `Debugging
Features`_.
5. A shortcoming of requiring a ``cocall`` keyword to call a coroutine
is that if is decided to implement coroutine-generators --
coroutines with ``yield`` or ``async yield`` expressions -- we
wouldn't need a ``cocall`` keyword to call them. So we'll end up
having ``__cocall__`` and no ``__call__`` for regular coroutines,
and having ``__call__`` and no ``__cocall__`` for coroutine-
generators.
6. Requiring parentheses grammatically also introduces a whole lot
of new problems.
The following code::
await fut
await function_returning_future()
await asyncio.gather(coro1(arg1, arg2), coro2(arg1, arg2))
would look like::
cocall fut() # or cocall costart(fut)
cocall (function_returning_future())()
cocall asyncio.gather(costart(coro1, arg1, arg2),
costart(coro2, arg1, arg2))
7. There are no equivalents of ``async for`` and ``async with`` in PEP
3152.
Coroutine-generators
--------------------
With ``async for`` keyword it is desirable to have a concept of a
*coroutine-generator* -- a coroutine with ``yield`` and ``yield from``
expressions. To avoid any ambiguity with regular generators, we would
likely require to have an ``async`` keyword before ``yield``, and
``async yield from`` would raise a ``StopAsyncIteration`` exception.
While it is possible to implement coroutine-generators, we believe that
they are out of scope of this proposal. It is an advanced concept that
should be carefully considered and balanced, with a non-trivial changes
in the implementation of current generator objects. This is a matter
for a separate PEP.
Why "async" and "await" keywords
--------------------------------
async/await is not a new concept in programming languages:
* C# has it since long time ago [5]_;
* proposal to add async/await in ECMAScript 7 [2]_;
see also Traceur project [9]_;
* Facebook's Hack/HHVM [6]_;
* Google's Dart language [7]_;
* Scala [8]_;
* proposal to add async/await to C++ [10]_;
* and many other less popular languages.
This is a huge benefit, as some users already have experience with
async/await, and because it makes working with many languages in one
project easier (Python with ECMAScript 7 for instance).
Why "__aiter__" returns awaitable
---------------------------------
In principle, ``__aiter__`` could be a regular function. There are
several good reasons to make it a coroutine:
* as most of the ``__anext__``, ``__aenter__``, and ``__aexit__``
methods are coroutines, users would often make a mistake defining it
as ``async`` anyways;
* there might be a need to run some asynchronous operations in
``__aiter__``, for instance to prepare DB queries or do some file
operation.
Importance of "async" keyword
-----------------------------
While it is possible to just implement ``await`` expression and treat
all functions with at least one ``await`` as coroutines, this approach
makes APIs design, code refactoring and its long time support harder.
Let's pretend that Python only has ``await`` keyword::
def useful():
...
await log(...)
...
def important():
await useful()
If ``useful()`` function is refactored and someone removes all
``await`` expressions from it, it would become a regular python
function, and all code that depends on it, including ``important()``
would be broken. To mitigate this issue a decorator similar to
``(a)asyncio.coroutine`` has to be introduced.
Why "async def"
---------------
For some people bare ``async name(): pass`` syntax might look more
appealing than ``async def name(): pass``. It is certainly easier to
type. But on the other hand, it breaks the symmetry between ``async
def``, ``async with`` and ``async for``, where ``async`` is a modifier,
stating that the statement is asynchronous. It is also more consistent
with the existing grammar.
Why not "await for" and "await with"
------------------------------------
``async`` is an adjective, and hence it is a better choice for a
*statement qualifier* keyword. ``await for/with`` would imply that
something is awaiting for a completion of a ``for`` or ``with``
statement.
Why "async def" and not "def async"
-----------------------------------
``async`` keyword is a *statement qualifier*. A good analogy to it are
"static", "public", "unsafe" keywords from other languages. "async
for" is an asynchronous "for" statement, "async with" is an
asynchronous "with" statement, "async def" is an asynchronous function.
Having "async" after the main statement keyword might introduce some
confusion, like "for async item in iterator" can be read as "for each
asynchronous item in iterator".
Having ``async`` keyword before ``def``, ``with`` and ``for`` also
makes the language grammar simpler. And "async def" better separates
coroutines from regular functions visually.
Why not a __future__ import
---------------------------
``__future__`` imports are inconvenient and easy to forget to add.
Also, they are enabled for the whole source file. Consider that there
is a big project with a popular module named "async.py". With future
imports it is required to either import it using ``__import__()`` or
``importlib.import_module()`` calls, or to rename the module. The
proposed approach makes it possible to continue using old code and
modules without a hassle, while coming up with a migration plan for
future python versions.
Why magic methods start with "a"
--------------------------------
New asynchronous magic methods ``__aiter__``, ``__anext__``,
``__aenter__``, and ``__aexit__`` all start with the same prefix "a".
An alternative proposal is to use "async" prefix, so that ``__aiter__``
becomes ``__async_iter__``. However, to align new magic methods with
the existing ones, such as ``__radd__`` and ``__iadd__`` it was decided
to use a shorter version.
Why not reuse existing magic names
----------------------------------
An alternative idea about new asynchronous iterators and context
managers was to reuse existing magic methods, by adding an ``async``
keyword to their declarations::
class CM:
async def __enter__(self): # instead of __aenter__
...
This approach has the following downsides:
* it would not be possible to create an object that works in both
``with`` and ``async with`` statements;
* it would break backwards compatibility, as nothing prohibits from
returning a Future-like objects from ``__enter__`` and/or
``__exit__`` in Python <= 3.4;
* one of the main points of this proposal is to make native coroutines
as simple and foolproof as possible, hence the clear separation of
the protocols.
Why not reuse existing "for" and "with" statements
--------------------------------------------------
The vision behind existing generator-based coroutines and this proposal
is to make it easy for users to see where the code might be suspended.
Making existing "for" and "with" statements to recognize asynchronous
iterators and context managers will inevitably create implicit suspend
points, making it harder to reason about the code.
Comprehensions
--------------
Syntax for asynchronous comprehensions could be provided, but this
construct is outside of the scope of this PEP.
Async lambda functions
----------------------
Syntax for asynchronous lambda functions could be provided, but this
construct is outside of the scope of this PEP.
Performance
===========
Overall Impact
--------------
This proposal introduces no observable performance impact. Here is an
output of python's official set of benchmarks [4]_:
::
python perf.py -r -b default ../cpython/python.exe
../cpython-aw/python.exe
[skipped]
Report on Darwin ysmac 14.3.0 Darwin Kernel Version 14.3.0:
Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64
x86_64 i386
Total CPU cores: 8
### etree_iterparse ###
Min: 0.365359 -> 0.349168: 1.05x faster
Avg: 0.396924 -> 0.379735: 1.05x faster
Significant (t=9.71)
Stddev: 0.01225 -> 0.01277: 1.0423x larger
The following not significant results are hidden, use -v to show them:
django_v2, 2to3, etree_generate, etree_parse, etree_process,
fastpickle,
fastunpickle, json_dump_v2, json_load, nbody, regex_v8, tornado_http.
Tokenizer modifications
-----------------------
There is no observable slowdown of parsing python files with the
modified tokenizer: parsing of one 12Mb file
(``Lib/test/test_binop.py`` repeated 1000 times) takes the same amount
of time.
async/await
-----------
The following micro-benchmark was used to determine performance
difference between "async" functions and generators::
import sys
import time
def binary(n):
if n <= 0:
return 1
l = yield from binary(n - 1)
r = yield from binary(n - 1)
return l + 1 + r
async def abinary(n):
if n <= 0:
return 1
l = await abinary(n - 1)
r = await abinary(n - 1)
return l + 1 + r
def timeit(gen, depth, repeat):
t0 = time.time()
for _ in range(repeat):
list(gen(depth))
t1 = time.time()
print('{}({}) * {}: total {:.3f}s'.format(
gen.__name__, depth, repeat, t1-t0))
The result is that there is no observable performance difference.
Minimum timing of 3 runs
::
abinary(19) * 30: total 12.985s
binary(19) * 30: total 12.953s
Note that depth of 19 means 1,048,575 calls.
Reference Implementation
========================
The reference implementation can be found here: [3]_.
List of high-level changes and new protocols
--------------------------------------------
1. New syntax for defining coroutines: ``async def`` and new ``await``
keyword.
2. New ``__await__`` method for Future-like objects, and new
``tp_await`` slot in ``PyTypeObject``.
3. New syntax for asynchronous context managers: ``async with``. And
associated protocol with ``__aenter__`` and ``__aexit__`` methods.
4. New syntax for asynchronous iteration: ``async for``. And
associated protocol with ``__aiter__``, ``__aexit__`` and new built-
in exception ``StopAsyncIteration``.
5. New AST nodes: ``AsyncFunctionDef``, ``AsyncFor``, ``AsyncWith``,
``Await``.
6. New functions: ``sys.set_coroutine_wrapper(callback)``,
``sys.get_coroutine_wrapper()``, ``types.coroutine(gen)``,
``inspect.iscoroutinefunction()``, ``inspect.iscoroutine()``,
and ``inspect.isawaitable()``.
7. New ``CO_COROUTINE`` and ``CO_NATIVE_COROUTINE`` bit flags for code
objects.
While the list of changes and new things is not short, it is important
to understand, that most users will not use these features directly.
It is intended to be used in frameworks and libraries to provide users
with convenient to use and unambiguous APIs with ``async def``,
``await``, ``async for`` and ``async with`` syntax.
Working example
---------------
All concepts proposed in this PEP are implemented [3]_ and can be
tested.
::
import asyncio
async def echo_server():
print('Serving on localhost:8000')
await asyncio.start_server(handle_connection,
'localhost', 8000)
async def handle_connection(reader, writer):
print('New connection...')
while True:
data = await reader.read(8192)
if not data:
break
print('Sending {:.10}... back'.format(repr(data)))
writer.write(data)
loop = asyncio.get_event_loop()
loop.run_until_complete(echo_server())
try:
loop.run_forever()
finally:
loop.close()
References
==========
.. [1] https://docs.python.org/3/library/asyncio-task.html#asyncio.coroutine
.. [2] http://wiki.ecmascript.org/doku.php?id=strawman:async_functions
.. [3] https://github.com/1st1/cpython/tree/await
.. [4] https://hg.python.org/benchmarks
.. [5] https://msdn.microsoft.com/en-us/library/hh191443.aspx
.. [6] http://docs.hhvm.com/manual/en/hack.async.php
.. [7] https://www.dartlang.org/articles/await-async/
.. [8] http://docs.scala-lang.org/sips/pending/async.html
.. [9]
https://github.com/google/traceur-compiler/wiki/LanguageFeatures#async-func…
.. [10]
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf (PDF)
.. [11]
https://docs.python.org/3/reference/expressions.html#generator-iterator-met…
.. [12] https://docs.python.org/3/reference/expressions.html#primaries
Acknowledgments
===============
I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Andrew
Svetlov, and Łukasz Langa for their initial feedback.
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
Hi python-dev,
Another round of updates. Reference implementation
has been updated: https://github.com/1st1/cpython/tree/await
(includes all things from the below summary of updates +
tests).
Summary:
1. "PyTypeObject.tp_await" slot. Replaces "tp_reserved".
This is to enable implementation of Futures with C API.
Must return an iterator if implemented.
2. New grammar for "await" expressions, see
'Syntax of "await" expression' section
3. inspect.iscoroutine() and inspect.iscoroutineobjects()
functions.
4. Full separation of coroutines and generators.
This is a big one; let's discuss.
a) Coroutine objects raise TypeError (is NotImplementedError
better?) in their __iter__ and __next__. Therefore it's
not not possible to pass them to iter(), tuple(), next() and
other similar functions that work with iterables.
b) Because of (a), for..in iteration also does not work
on coroutines anymore.
c) 'yield from' only accept coroutine objects from
generators decorated with 'types.coroutine'. That means
that existing asyncio generator-based coroutines will
happily yield from both coroutines and generators.
*But* every generator-based coroutine *must* be
decorated with `asyncio.coroutine()`. This is
potentially a backwards incompatible change.
d) inspect.isgenerator() and inspect.isgeneratorfunction()
return `False` for coroutine objects & coroutine functions.
e) Should we add a coroutine ABC (for cython etc)?
I, personally, think this is highly necessary. First,
separation of coroutines from generators is extremely
important. One day there won't be generator-based
coroutines, and we want to avoid any kind of confusion.
Second, we only can do this in 3.5. This kind of
semantics change won't be ever possible.
asyncio recommends using @coroutine decorator, and most
projects that I've seen do use it. Also there is no
reason for people to use iter() and next() functions
on coroutines when writing asyncio code. I doubt that
this will cause serious backwards compatibility problems
(asyncio also has provisional status).
Thank you,
Yury
PEP: 492
Title: Coroutines with async and await syntax
Version: $Revision$
Last-Modified: $Date$
Author: Yury Selivanov <yselivanov(a)sprymix.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 09-Apr-2015
Python-Version: 3.5
Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015
Abstract
========
This PEP introduces new syntax for coroutines, asynchronous ``with``
statements and ``for`` loops. The main motivation behind this proposal
is to streamline writing and maintaining asynchronous code, as well as
to simplify previously hard to implement code patterns.
Rationale and Goals
===================
Current Python supports implementing coroutines via generators (PEP
342), further enhanced by the ``yield from`` syntax introduced in PEP
380. This approach has a number of shortcomings:
* it is easy to confuse coroutines with regular generators, since they
share the same syntax; async libraries often attempt to alleviate
this by using decorators (e.g. ``(a)asyncio.coroutine`` [1]_);
* it is not possible to natively define a coroutine which has no
``yield`` or ``yield from`` statements, again requiring the use of
decorators to fix potential refactoring issues;
* support for asynchronous calls is limited to expressions where
``yield`` is allowed syntactically, limiting the usefulness of
syntactic features, such as ``with`` and ``for`` statements.
This proposal makes coroutines a native Python language feature, and
clearly separates them from generators. This removes
generator/coroutine ambiguity, and makes it possible to reliably define
coroutines without reliance on a specific library. This also enables
linters and IDEs to improve static code analysis and refactoring.
Native coroutines and the associated new syntax features make it
possible to define context manager and iteration protocols in
asynchronous terms. As shown later in this proposal, the new ``async
with`` statement lets Python programs perform asynchronous calls when
entering and exiting a runtime context, and the new ``async for``
statement makes it possible to perform asynchronous calls in iterators.
Specification
=============
This proposal introduces new syntax and semantics to enhance coroutine
support in Python, it does not change the internal implementation of
coroutines, which are still based on generators.
It is strongly suggested that the reader understands how coroutines are
implemented in Python (PEP 342 and PEP 380). It is also recommended to
read PEP 3156 (asyncio framework) and PEP 3152 (Cofunctions).
From this point in this document we use the word *coroutine* to refer
to functions declared using the new syntax. *generator-based
coroutine* is used where necessary to refer to coroutines that are
based on generator syntax.
New Coroutine Declaration Syntax
--------------------------------
The following new syntax is used to declare a coroutine::
async def read_data(db):
pass
Key properties of coroutines:
* ``async def`` functions are always coroutines, even if they do not
contain ``await`` expressions.
* It is a ``SyntaxError`` to have ``yield`` or ``yield from``
expressions in an ``async`` function.
* Internally, a new code object flag - ``CO_COROUTINE`` - is introduced
to enable runtime detection of coroutines (and migrating existing
code). All coroutines have both ``CO_COROUTINE`` and ``CO_GENERATOR``
flags set.
* Regular generators, when called, return a *generator object*;
similarly, coroutines return a *coroutine object*.
* ``StopIteration`` exceptions are not propagated out of coroutines,
and are replaced with a ``RuntimeError``. For regular generators
such behavior requires a future import (see PEP 479).
types.coroutine()
-----------------
A new function ``coroutine(gen)`` is added to the ``types`` module. It
applies ``CO_COROUTINE`` flag to the passed generator-function's code
object, making it to return a *coroutine object* when called.
This feature enables an easy upgrade path for existing libraries.
Await Expression
----------------
The following new ``await`` expression is used to obtain a result of
coroutine execution::
async def read_data(db):
data = await db.fetch('SELECT ...')
...
``await``, similarly to ``yield from``, suspends execution of
``read_data`` coroutine until ``db.fetch`` *awaitable* completes and
returns the result data.
It uses the ``yield from`` implementation with an extra step of
validating its argument. ``await`` only accepts an *awaitable*, which
can be one of:
* A *coroutine object* returned from a *coroutine* or a generator
decorated with ``types.coroutine()``.
* An object with an ``__await__`` method returning an iterator.
Any ``yield from`` chain of calls ends with a ``yield``. This is a
fundamental mechanism of how *Futures* are implemented. Since,
internally, coroutines are a special kind of generators, every
``await`` is suspended by a ``yield`` somewhere down the chain of
``await`` calls (please refer to PEP 3156 for a detailed
explanation.)
To enable this behavior for coroutines, a new magic method called
``__await__`` is added. In asyncio, for instance, to enable Future
objects in ``await`` statements, the only change is to add
``__await__ = __iter__`` line to ``asyncio.Future`` class.
Objects with ``__await__`` method are called *Future-like* objects in
the rest of this PEP.
Also, please note that ``__aiter__`` method (see its definition
below) cannot be used for this purpose. It is a different protocol,
and would be like using ``__iter__`` instead of ``__call__`` for
regular callables.
It is a ``TypeError`` if ``__await__`` returns anything but an
iterator.
* Objects defined with CPython C API with a ``tp_await`` function,
returning an iterator (similar to ``__await__`` method).
It is a ``SyntaxError`` to use ``await`` outside of a coroutine.
It is a ``TypeError`` to pass anything other than an *awaitable* object
to an ``await`` expression.
Syntax of "await" expression
''''''''''''''''''''''''''''
``await`` keyword is defined differently from ``yield`` and ``yield
from``. The main difference is that *await expressions* do not require
parentheses around them most of the times.
Examples::
================================== ==================================
Expression Will be parsed as
================================== ==================================
``if await fut: pass`` ``if (await fut): pass``
``if await fut + 1: pass`` ``if (await fut) + 1: pass``
``pair = await fut, 'spam'`` ``pair = (await fut), 'spam'``
``with await fut, open(): pass`` ``with (await fut), open(): pass``
``await foo()['spam'].baz()()`` ``await ( foo()['spam'].baz()() )``
``return await coro()`` ``return ( await coro() )``
``res = await coro() ** 2`` ``res = (await coro()) ** 2``
``func(a1=await coro(), a2=0)`` ``func(a1=(await coro()), a2=0)``
================================== ==================================
See `Grammar Updates`_ section for details.
Asynchronous Context Managers and "async with"
----------------------------------------------
An *asynchronous context manager* is a context manager that is able to
suspend execution in its *enter* and *exit* methods.
To make this possible, a new protocol for asynchronous context managers
is proposed. Two new magic methods are added: ``__aenter__`` and
``__aexit__``. Both must return an *awaitable*.
An example of an asynchronous context manager::
class AsyncContextManager:
async def __aenter__(self):
await log('entering context')
async def __aexit__(self, exc_type, exc, tb):
await log('exiting context')
New Syntax
''''''''''
A new statement for asynchronous context managers is proposed::
async with EXPR as VAR:
BLOCK
which is semantically equivalent to::
mgr = (EXPR)
aexit = type(mgr).__aexit__
aenter = type(mgr).__aenter__(mgr)
exc = True
try:
try:
VAR = await aenter
BLOCK
except:
exc = False
exit_res = await aexit(mgr, *sys.exc_info())
if not exit_res:
raise
finally:
if exc:
await aexit(mgr, None, None, None)
As with regular ``with`` statements, it is possible to specify multiple
context managers in a single ``async with`` statement.
It is an error to pass a regular context manager without ``__aenter__``
and ``__aexit__`` methods to ``async with``. It is a ``SyntaxError``
to use ``async with`` outside of a coroutine.
Example
'''''''
With asynchronous context managers it is easy to implement proper
database transaction managers for coroutines::
async def commit(session, data):
...
async with session.transaction():
...
await session.update(data)
...
Code that needs locking also looks lighter::
async with lock:
...
instead of::
with (yield from lock):
...
Asynchronous Iterators and "async for"
--------------------------------------
An *asynchronous iterable* is able to call asynchronous code in its
*iter* implementation, and *asynchronous iterator* can call
asynchronous code in its *next* method. To support asynchronous
iteration:
1. An object must implement an ``__aiter__`` method returning an
*awaitable* resulting in an *asynchronous iterator object*.
2. An *asynchronous iterator object* must implement an ``__anext__``
method returning an *awaitable*.
3. To stop iteration ``__anext__`` must raise a ``StopAsyncIteration``
exception.
An example of asynchronous iterable::
class AsyncIterable:
async def __aiter__(self):
return self
async def __anext__(self):
data = await self.fetch_data()
if data:
return data
else:
raise StopAsyncIteration
async def fetch_data(self):
...
New Syntax
''''''''''
A new statement for iterating through asynchronous iterators is
proposed::
async for TARGET in ITER:
BLOCK
else:
BLOCK2
which is semantically equivalent to::
iter = (ITER)
iter = await type(iter).__aiter__(iter)
running = True
while running:
try:
TARGET = await type(iter).__anext__(iter)
except StopAsyncIteration:
running = False
else:
BLOCK
else:
BLOCK2
It is a ``TypeError`` to pass a regular iterable without ``__aiter__``
method to ``async for``. It is a ``SyntaxError`` to use ``async for``
outside of a coroutine.
As for with regular ``for`` statement, ``async for`` has an optional
``else`` clause.
Example 1
'''''''''
With asynchronous iteration protocol it is possible to asynchronously
buffer data during iteration::
async for data in cursor:
...
Where ``cursor`` is an asynchronous iterator that prefetches ``N`` rows
of data from a database after every ``N`` iterations.
The following code illustrates new asynchronous iteration protocol::
class Cursor:
def __init__(self):
self.buffer = collections.deque()
def _prefetch(self):
...
async def __aiter__(self):
return self
async def __anext__(self):
if not self.buffer:
self.buffer = await self._prefetch()
if not self.buffer:
raise StopAsyncIteration
return self.buffer.popleft()
then the ``Cursor`` class can be used as follows::
async for row in Cursor():
print(row)
which would be equivalent to the following code::
i = await Cursor().__aiter__()
while True:
try:
row = await i.__anext__()
except StopAsyncIteration:
break
else:
print(row)
Example 2
'''''''''
The following is a utility class that transforms a regular iterable to
an asynchronous one. While this is not a very useful thing to do, the
code illustrates the relationship between regular and asynchronous
iterators.
::
class AsyncIteratorWrapper:
def __init__(self, obj):
self._it = iter(obj)
async def __aiter__(self):
return self
async def __anext__(self):
try:
value = next(self._it)
except StopIteration:
raise StopAsyncIteration
return value
async for letter in AsyncIteratorWrapper("abc"):
print(letter)
Why StopAsyncIteration?
'''''''''''''''''''''''
Coroutines are still based on generators internally. So, before PEP
479, there was no fundamental difference between
::
def g1():
yield from fut
return 'spam'
and
::
def g2():
yield from fut
raise StopIteration('spam')
And since PEP 479 is accepted and enabled by default for coroutines,
the following example will have its ``StopIteration`` wrapped into a
``RuntimeError``
::
async def a1():
await fut
raise StopIteration('spam')
The only way to tell the outside code that the iteration has ended is
to raise something other than ``StopIteration``. Therefore, a new
built-in exception class ``StopAsyncIteration`` was added.
Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions
raised in coroutines are wrapped in ``RuntimeError``.
Debugging Features
------------------
One of the most frequent mistakes that people make when using
generators as coroutines is forgetting to use ``yield from``::
@asyncio.coroutine
def useful():
asyncio.sleep(1) # this will do noting without 'yield from'
For debugging this kind of mistakes there is a special debug mode in
asyncio, in which ``@coroutine`` decorator wraps all functions with a
special object with a destructor logging a warning. Whenever a wrapped
generator gets garbage collected, a detailed logging message is
generated with information about where exactly the decorator function
was defined, stack trace of where it was collected, etc. Wrapper
object also provides a convenient ``__repr__`` function with detailed
information about the generator.
The only problem is how to enable these debug capabilities. Since
debug facilities should be a no-op in production mode, ``@coroutine``
decorator makes the decision of whether to wrap or not to wrap based on
an OS environment variable ``PYTHONASYNCIODEBUG``. This way it is
possible to run asyncio programs with asyncio's own functions
instrumented. ``EventLoop.set_debug``, a different debug facility, has
no impact on ``@coroutine`` decorator's behavior.
With this proposal, coroutines is a native, distinct from generators,
concept. New methods ``set_coroutine_wrapper`` and
``get_coroutine_wrapper`` are added to the ``sys`` module, with which
frameworks can provide advanced debugging facilities.
It is also important to make coroutines as fast and efficient as
possible, therefore there are no debug features enabled by default.
Example::
async def debug_me():
await asyncio.sleep(1)
def async_debug_wrap(generator):
return asyncio.CoroWrapper(generator)
sys.set_coroutine_wrapper(async_debug_wrap)
debug_me() # <- this line will likely GC the coroutine object and
# trigger asyncio.CoroWrapper's code.
assert isinstance(debug_me(), asyncio.CoroWrapper)
sys.set_coroutine_wrapper(None) # <- this unsets any
# previously set wrapper
assert not isinstance(debug_me(), asyncio.CoroWrapper)
If ``sys.set_coroutine_wrapper()`` is called twice, the new wrapper
replaces the previous wrapper. ``sys.set_coroutine_wrapper(None)``
unsets the wrapper.
inspect.iscoroutine() and inspect.iscoroutineobject()
-----------------------------------------------------
Two new functions are added to the ``inspect`` module:
* ``inspect.iscoroutine(obj)`` returns ``True`` if ``obj`` is a
coroutine object.
* ``inspect.iscoroutinefunction(obj)`` returns ``True`` is ``obj`` is a
coroutine function.
Differences between coroutines and generators
---------------------------------------------
A great effort has been made to make sure that coroutines and
generators are separate concepts:
1. Coroutine objects do not implement ``__iter__`` and ``__next__``
methods. Therefore they cannot be iterated over or passed to
``iter()``, ``list()``, ``tuple()`` and other built-ins. They
also cannot be used in a ``for..in`` loop.
2. ``yield from`` does not accept coroutine objects (unless it is used
in a generator-based coroutine decorated with ``types.coroutine``.)
3. ``yield from`` does not accept coroutine objects from plain Python
generators (*not* generator-based coroutines.)
4. ``inspect.isgenerator()`` and ``inspect.isgeneratorfunction()``
return ``False`` for coroutine objects and coroutine functions.
Coroutine objects
-----------------
Coroutines are based on generators internally, thus they share the
implementation. Similarly to generator objects, coroutine objects have
``throw``, ``send`` and ``close`` methods. ``StopIteration`` and
``GeneratorExit`` play the same role for coroutine objects (although
PEP 479 is enabled by default for coroutines).
Glossary
========
:Coroutine:
A coroutine function, or just "coroutine", is declared with ``async
def``. It uses ``await`` and ``return value``; see `New Coroutine
Declaration Syntax`_ for details.
:Coroutine object:
Returned from a coroutine function. See `Await Expression`_ for
details.
:Future-like object:
An object with an ``__await__`` method, or a C object with
``tp_await`` function, returning an iterator. Can be consumed by
an ``await`` expression in a coroutine. A coroutine waiting for a
Future-like object is suspended until the Future-like object's
``__await__`` completes, and returns the result. See `Await
Expression`_ for details.
:Awaitable:
A *Future-like* object or a *coroutine object*. See `Await
Expression`_ for details.
:Generator-based coroutine:
Coroutines based in generator syntax. Most common example is
``(a)asyncio.coroutine``.
:Asynchronous context manager:
An asynchronous context manager has ``__aenter__`` and ``__aexit__``
methods and can be used with ``async with``. See `Asynchronous
Context Managers and "async with"`_ for details.
:Asynchronous iterable:
An object with an ``__aiter__`` method, which must return an
*asynchronous iterator* object. Can be used with ``async for``.
See `Asynchronous Iterators and "async for"`_ for details.
:Asynchronous iterator:
An asynchronous iterator has an ``__anext__`` method. See
`Asynchronous Iterators and "async for"`_ for details.
List of functions and methods
=============================
================= =================================== =================
Method Can contain Can't contain
================= =================================== =================
async def func await, return value yield, yield from
async def __a*__ await, return value yield, yield from
def __a*__ return awaitable await
def __await__ yield, yield from, return iterable await
generator yield, yield from, return value await
================= =================================== =================
Where:
* "async def func": coroutine;
* "async def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``,
``__aexit__`` defined with the ``async`` keyword;
* "def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``,
``__aexit__`` defined without the ``async`` keyword, must return an
*awaitable*;
* "def __await__": ``__await__`` method to implement *Future-like*
objects;
* generator: a "regular" generator, function defined with ``def`` and
which contains a least one ``yield`` or ``yield from`` expression.
Transition Plan
===============
To avoid backwards compatibility issues with ``async`` and ``await``
keywords, it was decided to modify ``tokenizer.c`` in such a way, that
it:
* recognizes ``async def`` name tokens combination (start of a
coroutine);
* keeps track of regular functions and coroutines;
* replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with
``AWAIT`` when in the process of yielding tokens for coroutines.
This approach allows for seamless combination of new syntax features
(all of them available only in ``async`` functions) with any existing
code.
An example of having "async def" and "async" attribute in one piece of
code::
class Spam:
async = 42
async def ham():
print(getattr(Spam, 'async'))
# The coroutine can be executed and will print '42'
Backwards Compatibility
-----------------------
This proposal preserves 100% backwards compatibility.
Grammar Updates
---------------
Grammar changes are also fairly minimal::
decorated: decorators (classdef | funcdef | async_funcdef)
async_funcdef: ASYNC funcdef
compound_stmt: (if_stmt | while_stmt | for_stmt | try_stmt | with_stmt
| funcdef | classdef | decorated | async_stmt)
async_stmt: ASYNC (funcdef | with_stmt | for_stmt)
power: atom_expr ['**' factor]
atom_expr: [AWAIT] atom trailer*
Transition Period Shortcomings
------------------------------
There is just one.
Until ``async`` and ``await`` are not proper keywords, it is not
possible (or at least very hard) to fix ``tokenizer.c`` to recognize
them on the **same line** with ``def`` keyword::
# async and await will always be parsed as variables
async def outer(): # 1
def nested(a=(await fut)):
pass
async def foo(): return (await fut) # 2
Since ``await`` and ``async`` in such cases are parsed as ``NAME``
tokens, a ``SyntaxError`` will be raised.
To workaround these issues, the above examples can be easily rewritten
to a more readable form::
async def outer(): # 1
a_default = await fut
def nested(a=a_default):
pass
async def foo(): # 2
return (await fut)
This limitation will go away as soon as ``async`` and ``await`` ate
proper keywords. Or if it's decided to use a future import for this
PEP.
Deprecation Plans
-----------------
``async`` and ``await`` names will be softly deprecated in CPython 3.5
and 3.6. In 3.7 we will transform them to proper keywords. Making
``async`` and ``await`` proper keywords before 3.7 might make it harder
for people to port their code to Python 3.
asyncio
-------
``asyncio`` module was adapted and tested to work with coroutines and
new statements. Backwards compatibility is 100% preserved.
The required changes are mainly:
1. Modify ``(a)asyncio.coroutine`` decorator to use new
``types.coroutine()`` function.
2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class.
3. Add ``ensure_task()`` as an alias for ``async()`` function.
Deprecate ``async()`` function.
Design Considerations
=====================
PEP 3152
--------
PEP 3152 by Gregory Ewing proposes a different mechanism for coroutines
(called "cofunctions"). Some key points:
1. A new keyword ``codef`` to declare a *cofunction*. *Cofunction* is
always a generator, even if there is no ``cocall`` expressions
inside it. Maps to ``async def`` in this proposal.
2. A new keyword ``cocall`` to call a *cofunction*. Can only be used
inside a *cofunction*. Maps to ``await`` in this proposal (with
some differences, see below.)
3. It is not possible to call a *cofunction* without a ``cocall``
keyword.
4. ``cocall`` grammatically requires parentheses after it::
atom: cocall | <existing alternatives for atom>
cocall: 'cocall' atom cotrailer* '(' [arglist] ')'
cotrailer: '[' subscriptlist ']' | '.' NAME
5. ``cocall f(*args, **kwds)`` is semantically equivalent to
``yield from f.__cocall__(*args, **kwds)``.
Differences from this proposal:
1. There is no equivalent of ``__cocall__`` in this PEP, which is
called and its result is passed to ``yield from`` in the ``cocall``
expression. ``await`` keyword expects an *awaitable* object,
validates the type, and executes ``yield from`` on it. Although,
``__await__`` method is similar to ``__cocall__``, but is only used
to define *Future-like* objects.
2. ``await`` is defined in almost the same way as ``yield from`` in the
grammar (it is later enforced that ``await`` can only be inside
``async def``). It is possible to simply write ``await future``,
whereas ``cocall`` always requires parentheses.
3. To make asyncio work with PEP 3152 it would be required to modify
``(a)asyncio.coroutine`` decorator to wrap all functions in an object
with a ``__cocall__`` method, or to implement ``__cocall__`` on
generators. To call *cofunctions* from existing generator-based
coroutines it would be required to use ``costart(cofunc, *args,
**kwargs)`` built-in.
4. Since it is impossible to call a *cofunction* without a ``cocall``
keyword, it automatically prevents the common mistake of forgetting
to use ``yield from`` on generator-based coroutines. This proposal
addresses this problem with a different approach, see `Debugging
Features`_.
5. A shortcoming of requiring a ``cocall`` keyword to call a coroutine
is that if is decided to implement coroutine-generators --
coroutines with ``yield`` or ``async yield`` expressions -- we
wouldn't need a ``cocall`` keyword to call them. So we'll end up
having ``__cocall__`` and no ``__call__`` for regular coroutines,
and having ``__call__`` and no ``__cocall__`` for coroutine-
generators.
6. Requiring parentheses grammatically also introduces a whole lot
of new problems.
The following code::
await fut
await function_returning_future()
await asyncio.gather(coro1(arg1, arg2), coro2(arg1, arg2))
would look like::
cocall fut() # or cocall costart(fut)
cocall (function_returning_future())()
cocall asyncio.gather(costart(coro1, arg1, arg2),
costart(coro2, arg1, arg2))
7. There are no equivalents of ``async for`` and ``async with`` in PEP
3152.
Coroutine-generators
--------------------
With ``async for`` keyword it is desirable to have a concept of a
*coroutine-generator* -- a coroutine with ``yield`` and ``yield from``
expressions. To avoid any ambiguity with regular generators, we would
likely require to have an ``async`` keyword before ``yield``, and
``async yield from`` would raise a ``StopAsyncIteration`` exception.
While it is possible to implement coroutine-generators, we believe that
they are out of scope of this proposal. It is an advanced concept that
should be carefully considered and balanced, with a non-trivial changes
in the implementation of current generator objects. This is a matter
for a separate PEP.
No implicit wrapping in Futures
-------------------------------
There is a proposal to add similar mechanism to ECMAScript 7 [2]_. A
key difference is that JavaScript "async functions" always return a
Promise. While this approach has some advantages, it also implies that
a new Promise object is created on each "async function" invocation.
We could implement a similar functionality in Python, by wrapping all
coroutines in a Future object, but this has the following
disadvantages:
1. Performance. A new Future object would be instantiated on each
coroutine call. Moreover, this makes implementation of ``await``
expressions slower (disabling optimizations of ``yield from``).
2. A new built-in ``Future`` object would need to be added.
3. Coming up with a generic ``Future`` interface that is usable for any
use case in any framework is a very hard to solve problem.
4. It is not a feature that is used frequently, when most of the code
is coroutines.
Why "async" and "await" keywords
--------------------------------
async/await is not a new concept in programming languages:
* C# has it since long time ago [5]_;
* proposal to add async/await in ECMAScript 7 [2]_;
see also Traceur project [9]_;
* Facebook's Hack/HHVM [6]_;
* Google's Dart language [7]_;
* Scala [8]_;
* proposal to add async/await to C++ [10]_;
* and many other less popular languages.
This is a huge benefit, as some users already have experience with
async/await, and because it makes working with many languages in one
project easier (Python with ECMAScript 7 for instance).
Why "__aiter__" is a coroutine
------------------------------
In principle, ``__aiter__`` could be a regular function. There are
several good reasons to make it a coroutine:
* as most of the ``__anext__``, ``__aenter__``, and ``__aexit__``
methods are coroutines, users would often make a mistake defining it
as ``async`` anyways;
* there might be a need to run some asynchronous operations in
``__aiter__``, for instance to prepare DB queries or do some file
operation.
Importance of "async" keyword
-----------------------------
While it is possible to just implement ``await`` expression and treat
all functions with at least one ``await`` as coroutines, this approach
makes APIs design, code refactoring and its long time support harder.
Let's pretend that Python only has ``await`` keyword::
def useful():
...
await log(...)
...
def important():
await useful()
If ``useful()`` function is refactored and someone removes all
``await`` expressions from it, it would become a regular python
function, and all code that depends on it, including ``important()``
would be broken. To mitigate this issue a decorator similar to
``(a)asyncio.coroutine`` has to be introduced.
Why "async def"
---------------
For some people bare ``async name(): pass`` syntax might look more
appealing than ``async def name(): pass``. It is certainly easier to
type. But on the other hand, it breaks the symmetry between ``async
def``, ``async with`` and ``async for``, where ``async`` is a modifier,
stating that the statement is asynchronous. It is also more consistent
with the existing grammar.
Why "async for/with" instead of "await for/with"
------------------------------------------------
``async`` is an adjective, and hence it is a better choice for a
*statement qualifier* keyword. ``await for/with`` would imply that
something is awaiting for a completion of a ``for`` or ``with``
statement.
Why "async def" and not "def async"
-----------------------------------
``async`` keyword is a *statement qualifier*. A good analogy to it are
"static", "public", "unsafe" keywords from other languages. "async
for" is an asynchronous "for" statement, "async with" is an
asynchronous "with" statement, "async def" is an asynchronous function.
Having "async" after the main statement keyword might introduce some
confusion, like "for async item in iterator" can be read as "for each
asynchronous item in iterator".
Having ``async`` keyword before ``def``, ``with`` and ``for`` also
makes the language grammar simpler. And "async def" better separates
coroutines from regular functions visually.
Why not a __future__ import
---------------------------
``__future__`` imports are inconvenient and easy to forget to add.
Also, they are enabled for the whole source file. Consider that there
is a big project with a popular module named "async.py". With future
imports it is required to either import it using ``__import__()`` or
``importlib.import_module()`` calls, or to rename the module. The
proposed approach makes it possible to continue using old code and
modules without a hassle, while coming up with a migration plan for
future python versions.
Why magic methods start with "a"
--------------------------------
New asynchronous magic methods ``__aiter__``, ``__anext__``,
``__aenter__``, and ``__aexit__`` all start with the same prefix "a".
An alternative proposal is to use "async" prefix, so that ``__aiter__``
becomes ``__async_iter__``. However, to align new magic methods with
the existing ones, such as ``__radd__`` and ``__iadd__`` it was decided
to use a shorter version.
Why not reuse existing magic names
----------------------------------
An alternative idea about new asynchronous iterators and context
managers was to reuse existing magic methods, by adding an ``async``
keyword to their declarations::
class CM:
async def __enter__(self): # instead of __aenter__
...
This approach has the following downsides:
* it would not be possible to create an object that works in both
``with`` and ``async with`` statements;
* it would break backwards compatibility, as nothing prohibits from
returning a Future-like objects from ``__enter__`` and/or
``__exit__`` in Python <= 3.4;
* one of the main points of this proposal is to make coroutines as
simple and foolproof as possible, hence the clear separation of the
protocols.
Why not reuse existing "for" and "with" statements
--------------------------------------------------
The vision behind existing generator-based coroutines and this proposal
is to make it easy for users to see where the code might be suspended.
Making existing "for" and "with" statements to recognize asynchronous
iterators and context managers will inevitably create implicit suspend
points, making it harder to reason about the code.
Comprehensions
--------------
For the sake of restricting the broadness of this PEP there is no new
syntax for asynchronous comprehensions. This should be considered in a
separate PEP, if there is a strong demand for this feature.
Async lambdas
-------------
Lambda coroutines are not part of this proposal. In this proposal they
would look like ``async lambda(parameters): expression``. Unless there
is a strong demand to have them as part of this proposal, it is
recommended to consider them later in a separate PEP.
Performance
===========
Overall Impact
--------------
This proposal introduces no observable performance impact. Here is an
output of python's official set of benchmarks [4]_:
::
python perf.py -r -b default ../cpython/python.exe
../cpython-aw/python.exe
[skipped]
Report on Darwin ysmac 14.3.0 Darwin Kernel Version 14.3.0:
Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64
x86_64 i386
Total CPU cores: 8
### etree_iterparse ###
Min: 0.365359 -> 0.349168: 1.05x faster
Avg: 0.396924 -> 0.379735: 1.05x faster
Significant (t=9.71)
Stddev: 0.01225 -> 0.01277: 1.0423x larger
The following not significant results are hidden, use -v to show them:
django_v2, 2to3, etree_generate, etree_parse, etree_process,
fastpickle,
fastunpickle, json_dump_v2, json_load, nbody, regex_v8, tornado_http.
Tokenizer modifications
-----------------------
There is no observable slowdown of parsing python files with the
modified tokenizer: parsing of one 12Mb file
(``Lib/test/test_binop.py`` repeated 1000 times) takes the same amount
of time.
async/await
-----------
The following micro-benchmark was used to determine performance
difference between "async" functions and generators::
import sys
import time
def binary(n):
if n <= 0:
return 1
l = yield from binary(n - 1)
r = yield from binary(n - 1)
return l + 1 + r
async def abinary(n):
if n <= 0:
return 1
l = await abinary(n - 1)
r = await abinary(n - 1)
return l + 1 + r
def timeit(gen, depth, repeat):
t0 = time.time()
for _ in range(repeat):
list(gen(depth))
t1 = time.time()
print('{}({}) * {}: total {:.3f}s'.format(
gen.__name__, depth, repeat, t1-t0))
The result is that there is no observable performance difference.
Minimum timing of 3 runs
::
abinary(19) * 30: total 12.985s
binary(19) * 30: total 12.953s
Note that depth of 19 means 1,048,575 calls.
Reference Implementation
========================
The reference implementation can be found here: [3]_.
List of high-level changes and new protocols
--------------------------------------------
1. New syntax for defining coroutines: ``async def`` and new ``await``
keyword.
2. New ``__await__`` method for Future-like objects, and new
``tp_await`` slot in ``PyTypeObject``.
3. New syntax for asynchronous context managers: ``async with``. And
associated protocol with ``__aenter__`` and ``__aexit__`` methods.
4. New syntax for asynchronous iteration: ``async for``. And
associated protocol with ``__aiter__``, ``__aexit__`` and new built-
in exception ``StopAsyncIteration``.
5. New AST nodes: ``AsyncFunctionDef``, ``AsyncFor``, ``AsyncWith``,
``Await``.
6. New functions: ``sys.set_coroutine_wrapper(callback)``,
``sys.get_coroutine_wrapper()``, ``types.coroutine(gen)``,
``inspect.iscoroutinefunction()``, and ``inspect.iscoroutine()``.
7. New ``CO_COROUTINE`` bit flag for code objects.
While the list of changes and new things is not short, it is important
to understand, that most users will not use these features directly.
It is intended to be used in frameworks and libraries to provide users
with convenient to use and unambiguous APIs with ``async def``,
``await``, ``async for`` and ``async with`` syntax.
Working example
---------------
All concepts proposed in this PEP are implemented [3]_ and can be
tested.
::
import asyncio
async def echo_server():
print('Serving on localhost:8000')
await asyncio.start_server(handle_connection,
'localhost', 8000)
async def handle_connection(reader, writer):
print('New connection...')
while True:
data = await reader.read(8192)
if not data:
break
print('Sending {:.10}... back'.format(repr(data)))
writer.write(data)
loop = asyncio.get_event_loop()
loop.run_until_complete(echo_server())
try:
loop.run_forever()
finally:
loop.close()
References
==========
.. [1] https://docs.python.org/3/library/asyncio-task.html#asyncio.coroutine
.. [2] http://wiki.ecmascript.org/doku.php?id=strawman:async_functions
.. [3] https://github.com/1st1/cpython/tree/await
.. [4] https://hg.python.org/benchmarks
.. [5] https://msdn.microsoft.com/en-us/library/hh191443.aspx
.. [6] http://docs.hhvm.com/manual/en/hack.async.php
.. [7] https://www.dartlang.org/articles/await-async/
.. [8] http://docs.scala-lang.org/sips/pending/async.html
.. [9]
https://github.com/google/traceur-compiler/wiki/LanguageFeatures#async-func…
.. [10]
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf (PDF)
Acknowledgments
===============
I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Andrew
Svetlov, and Łukasz Langa for their initial feedback.
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
I've tried to catch up with the previous threads. A summary of issues
brought up:
1. precise syntax of `async def` (or do we need it at all)
2. do we need `async for` and `async with` (and how to spell them)
3. syntactic priority of `await`
4. `cocall` vs. `await`
5. do we really need `__aiter__` and friends
6. StopAsyncException
7. compatibility with asyncio and existing users of it
(I've added a few myself.)
I'll try to take them one by one.
*1. precise syntax of `async def`*
Of all the places to put `async` I still like *before* the `def` the best.
I often do "imprecise search" for e.g. /def foo/ and would be unhappy if
this didn't find async defs. Putting it towards the end (`def foo async()`
or `def foo() async`) makes it easier to miss. A decorator makes it hard to
make the syntactic distinctions required to reject `await` outside an async
function. So I still prefer *`async def`*.
*2. do we need `async for` and `async with`*
Yes we do. Most of you are too young to remember, but once upon a time you
couldn't loop over the lines of a file with a `for` loop like you do now.
The amount of code that was devoted to efficiently iterate over files was
tremendous. We're back in that stone age with the asyncio `StreamReader`
class -- it supports `read()`, `readline()` and so on, but you can't use it
with `for`, so you have to write a `while True` loop. `asyncio for` makes
it possible to add a simple `__anext__` to the `StreamReader` class, as
follows:
```
async def __anext__(self):
line = await self.readline()
if not line:
raise StopAsyncIteration
return line
```
A similar argument can be made for `async with`; the transaction commit is
pretty convincing, but it also helps to be able to wait e.g. for a
transport to drain upon closing a write stream. As for how to spell these,
I think having `async` at the front makes it most clear that this is a
special form.
(Though maybe we should consider `await for` and `await with`? That would
have the advantage of making it easy to scan for all suspension points by
searching for /await/. But being a verb it doesn't read very well.)
*3. syntactic priority of `await`*
Yury, could you tweak the syntax for `await` so that we can write the most
common usages without parentheses? In particular I'd like to be able to
write
```
return await foo()
with await foo() as bar: ...
foo(await bar(), await bletch())
```
(I don't care about `await foo() + await bar()` but it would be okay.)
```
I think this is reasonable with some tweaks of the grammar (similar to what
Greg did for cocall, but without requiring call syntax at the end).
*4. `cocall` vs. `await`*
Python evolves. We couldn't have PEP 380 (`yield from`) without prior
experience with using generators as coroutines (PEP 342), which in turn
required basic generators (PEP 255), and those were a natural evolution of
Python's earlier `for` loop.
We couldn't PEP 3156 (asyncio) without PEP 380 and all that came before.
The asyncio library is getting plenty of adoption and it has the concept of
separating the *getting* of a future[1] from *waiting* for it. IIUC this
is also how `await` works in C# (it just requires something with an async
type). This has enabled a variety of operations that take futures and
produce more futures.
[1] I write `future` with a lowercase 'f' to include concepts like
coroutine generator objects.
*I just can't get used to this aspect of PEP 3152, so I'm rejecting it.*
Sorry Greg, but that's the end. We must see `await` as a refinement of
`yield from`, not as an alternative. (Yury: PEP 492 is not accepted yet,
but you're getting closer.)
One more thing: this separation is "Pythonic" in the sense that it's
similar to the way *getting* a callable object is a separate act from
*calling* it. While this is a cause for newbie bugs (forgetting to call an
argument-less function) it has also enabled the concept of "callable" as
more general and more powerful in Python: any time you need to pass a
callable, you can pass e.g. a bound method or a class or something you got
from `functools.partial`, and that's a useful thing (other languages
require you to introduce something like a lambda in such cases, which can
be painful if the thing you wrap has a complex signature -- or they don't
support function parameters at all, like Java).
I know that Greg defends it by explaining that `cocal f(args)` is not a
`cocall` operator applied to `f(args)`, it is the *single* operator `cocall
...(args)` applied to `f`. But this is too subtle, and it just doesn't jive
with the long tradition of using `yield from f` where f is some previously
obtained future.
*5. do we really need `__aiter__` and friends*
There's a lot of added complexity, but I think it's worth it. I don't think
we need to make the names longer, the 'a' prefix is fine for these methods.
I think it's all in the protocols: regular `with` uses `__enter__` and
`__exit__`; `async with` uses `__aenter__` and `__aexit__` (which must
return futures).
Ditto for `__aiter__` and `__anext__`. I guess this means that the async
equivalent to obtaining an iterator through `it = iter(xs)` followed by
`for x over it` will have to look like `ait = await aiter(xs)` followed by
`for x over ait`, where an iterator is required to have an `__aiter__`
method that's an async function and returns self immediately. But what if
you left out the `await` from the first call? I.e. can this work?
```
ait = aiter(xs)
async for x in ait:
print(x)
```
The question here is whether the object returned by aiter(xs) has an
`__aiter__` method. Since it was intended to be the target of `await`, it
has an `__await__` method. But that itself is mostly an alias for
`__iter__`, not `__aiter__`. I guess it can be made to work, the object
just has to implement a bunch of different protocols.
*6. StopAsyncException*
I'm not sure about this. The motivation given in the PEP seems to focus on
the need for `__anext__` to be async. But is this really the right pattern?
What if we required `ait.__anext__()` to return a future, which can either
raise good old `StopIteration` or return the next value from the iteration
when awaited? I'm wondering if there are a few alternatives to be explored
around the async iterator protocol still.
*7. compatibility with asyncio and existing users of it*
This is just something I want to stress. On the one hand it should be
really simple to take code written for pre-3.5 asyncio and rewrite it to
PEP 492 -- simple change `(a)asyncio.coroutine` to `async def` and change
`yield from` to `await`. (Everything else is optional; existing patterns
for loops and context managers should continue to work unchanged, even if
in some cases you may be able to simplify the code by using `async for` and
`async with`.)
But it's also important that *even if you don't switch* (i.e. if you want
to keep your code compatible with pre-3.5 asyncio) you can still use the
PEP 492 version of asyncio -- i.e. the asyncio library that comes with 3.5
must seamlessly support mixing code that uses `await` and code that uses
`yield from`. And this should go both ways -- if you have some code that
uses PEP 492 and some code that uses pre-3.5 asyncio, they should be able
to pass their coroutines to each other and wait for each other's coroutines.
That's all I have for now. Enjoy!
--
--Guido van Rossum (python.org/~guido)
Real world stuff is devouring my free time since immediately after PyCon
and will continue to do so for probably the next few months. I'm hoping to
find the energy to engage Donald and Nick about their proposals while I'm
time-constrained so that when I do have free time again I will be able to
make a decision quickly.
This also means that I'm allowing Donald and Nick to update their PEPs
while they wait for me, although I'm not taking on any more proposals as
the two current proposals cover the two ranges of suggestions people have
talked to me about on this topic.
My apologies to Nick and Donald for slipping on my own deadline.
