Python-Dev
Threads by month
- ----- 2025 -----
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2018 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2017 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2016 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2015 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2014 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2013 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2012 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2011 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2010 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2009 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2008 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2007 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2006 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2005 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2004 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2003 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2002 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2001 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2000 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 1999 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
May 2015
- 112 participants
- 87 discussions
Here is a statistic for most called PyObject_INIT or PyObject_INIT_VAR
for types (collected during running Python tests on 32-bit Linux).
type count % acc.%
builtin_function_or_method 116012007 36.29% 36.29%
method 52465386 16.41% 52.70%
int 42828741 13.40% 66.09%
str 37017098 11.58% 77.67%
generator 14026583 4.39% 82.06%
…
[View More]list_iterator 8731329 2.73% 84.79%
bytes 7217934 2.26% 87.04%
tuple_iterator 5042563 1.58% 88.62%
float 4672980 1.46% 90.08%
set 3319699 1.04% 91.12%
_io.StringIO 3000369 0.94% 92.06%
str_iterator 2126838 0.67% 92.73%
list 2031059 0.64% 93.36%
dict 1691993 0.53% 93.89%
method-wrapper 1573139 0.49% 94.38%
function 1472062 0.46% 94.84%
traceback 1388278 0.43% 95.28%
tuple 1132071 0.35% 95.63%
memoryview 1092173 0.34% 95.97%
cell 1049496 0.33% 96.30%
managedbuffer 1036889 0.32% 96.63%
bytearray 711969 0.22% 96.85%
range_iterator 496924 0.16% 97.00%
range 483971 0.15% 97.15%
super 472447 0.15% 97.30%
map 449567 0.14% 97.44%
frame 427320 0.13% 97.58%
set_iterator 423392 0.13% 97.71%
Leaf 398705 0.12% 97.83%
symtable 374412 0.12% 97.95%
Types for which free lists already are used: builtin_function_or_method,
method, float, tuple, list, dict, frame. Some free list implementations
(e.g. for tuple) don't call PyObject_INIT/PyObject_INIT_VAR. That is why
numbers are such low for tuples.
Perhaps it is worth to add free lists for other types: int, str, bytes,
generator, list and tuple iterators?
Shortened tables for variable-sized objects (that calls PyObject_INIT_VAR):
int 42828741 13.40%
0 425353 0.99% 0.99%
1 21399290 49.96% 50.96%
2 10496856 24.51% 75.47%
3 4873346 11.38% 86.85%
4 1021563 2.39% 89.23%
5 1246444 2.91% 92.14%
6 733676 1.71% 93.85%
7 123074 0.29% 94.14%
8 139203 0.33% 94.47%
...
bytes 7217934 2.26%
0 842 0.01% 0.01%
1 179469 2.49% 2.50%
2 473306 6.56% 9.06%
3 254968 3.53% 12.59%
4 1169164 16.20% 28.79%
5 72806 1.01% 29.79%
6 128668 1.78% 31.58%
7 169694 2.35% 33.93%
8 155154 2.15% 36.08%
9 67320 0.93% 37.01%
10 51703 0.72% 37.73%
11 42574 0.59% 38.32%
12 108947 1.51% 39.83%
13 40812 0.57% 40.39%
14 126783 1.76% 42.15%
15 37873 0.52% 42.67%
16 447482 6.20% 48.87%
17 194320 2.69% 51.56%
18 251685 3.49% 55.05%
19 159435 2.21% 57.26%
20 212521 2.94% 60.20%
...
31 18751 0.26% 67.32%
32 159781 2.21% 69.54%
33 8332 0.12% 69.65%
...
63 19841 0.27% 79.21%
64 144982 2.01% 81.22%
65 5216 0.07% 81.29%
...
127 1354 0.02% 85.44%
128 376539 5.22% 90.66%
129 17468 0.24% 90.90%
...
255 178 0.00% 92.39%
256 11993 0.17% 92.55%
257 124 0.00% 92.56%
...
[View Less]
4
7

Re: [Python-Dev] [Python-checkins] cpython (3.4): asyncio: async() function is deprecated in favour of ensure_future().
by Paul Moore May 11, 2015
by Paul Moore May 11, 2015
May 11, 2015
On 11 May 2015 at 19:50, yury.selivanov <python-checkins(a)python.org> wrote:
> https://hg.python.org/cpython/rev/b78127eafad7
> changeset: 95956:b78127eafad7
> branch: 3.4
> parent: 95953:a983d63e3321
> user: Yury Selivanov <yselivanov(a)sprymix.com>
> date: Mon May 11 14:48:38 2015 -0400
> summary:
> asyncio: async() function is deprecated in favour of ensure_future().
>
> files:
> Lib/asyncio/base_events.py …
[View More] | 2 +-
> Lib/asyncio/tasks.py | 27 ++++-
> Lib/asyncio/windows_events.py | 2 +-
> Lib/test/test_asyncio/test_base_events.py | 6 +-
> Lib/test/test_asyncio/test_tasks.py | 48 +++++----
> Lib/test/test_asyncio/test_windows_events.py | 2 +-
> Misc/NEWS | 4 +-
> 7 files changed, 57 insertions(+), 34 deletions(-)
Surely this should include a doc change?
Paul
[View Less]
2
2
It is my privilege to announce the first release candidate of 2.7.10,
the next bugfix release in the 2.7 series.
Downloads are at
https://www.python.org/downloads/release/python-2710rc1/
The full changelog is at
https://hg.python.org/cpython/raw-file/80ccce248ba2/Misc/NEWS
Please consider testing 2.7.10rc1 with your application and reporting
bugs to
https://bugs.python.org
Regards,
Benjamin
1
0
Here's something that might be wrong in Python (tried on v2.7):
>>> class int(str): pass
>>> int(3)
'3'
Mark
2
1
In Python's argument parsing code (convertsimple in Python/getargs.c), a
couple of format units* accept "read-only bytes-like objects", aka
read-only buffer objects. They call a helper function called
convertbuffer() which uses the buffer protocol to extract a pointer to
the memory.
Here's the relevant bit of code:
static Py_ssize_t
convertbuffer(PyObject *arg, void **p, char **errmsg)
{
Py_buffer view;
...
if (getbuffer(arg, &view, errmsg) < 0)
…
[View More]return -1;
count = view.len;
*p = view.buf;
PyBuffer_Release(&view);
return count;
}
getbuffer() uses the buffer protocol to fill in the "view" buffer. If
it's successful, "view" is a valid buffer. We store the pointer to the
buffer's memory in output parameter p.
THEN WE RELEASE THE BUFFER.
THEN WE RETURN TO THE CALLER.
In case you missed the big helpful capital letters, we are returning a
pointer given to us by PyObject_GetBuffer(), which we have already
released by calling PyBuffer_Release(). The buffer protocol
documentation for bf_releasebuffer makes it sound like this pointer
could easily be invalid after the release call finishes.
Am I missing something, or is this code relying on an implementation
detail it shouldn't--namely that you can continue using a pointer to
some (most? all?) buffer memory even after releasing it?
//arry/
* Specifically: s# y y# z#
[View Less]
2
1
In the spirit of regular releases, it's time to release 2.7.10. I'm
going to plan to cut rc1 this weekend with a final in 2 weeks.
I apologize for the short notice; time has crept up on me, and I have
commitments in June that prevent pushing releases into that month.
1
0
ACTIVITY SUMMARY (2015-05-01 - 2015-05-08)
Python tracker at http://bugs.python.org/
To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.
Issues counts and deltas:
open 4838 ( -3)
closed 31069 (+44)
total 35907 (+41)
Open issues with patches: 2252
Issues opened (25)
==================
#24107: Add support for retrieving the certificate chain
http://bugs.python.org/issue24107 opened by Lukasa
#24109: Documentation for difflib …
[View More]uses optparse
http://bugs.python.org/issue24109 opened by idahogray
#24110: zipfile.ZipFile.write() does not accept bytes arcname
http://bugs.python.org/issue24110 opened by july
#24111: Valgrind suppression file should be updated
http://bugs.python.org/issue24111 opened by Antony.Lee
#24114: ctypes.utils uninitialized variable 'path'
http://bugs.python.org/issue24114 opened by kees
#24115: PyObject_IsInstance() and PyObject_IsSubclass() can fail
http://bugs.python.org/issue24115 opened by serhiy.storchaka
#24116: --with-pydebug has no effect when the final python binary is c
http://bugs.python.org/issue24116 opened by aleb
#24117: Wrong range checking in GB18030 decoder.
http://bugs.python.org/issue24117 opened by Ma Lin
#24119: Carry comments with the AST
http://bugs.python.org/issue24119 opened by brett.cannon
#24120: pathlib.(r)glob stops on PermissionDenied exception
http://bugs.python.org/issue24120 opened by Gregorio
#24124: Two versions of instructions for installing Python modules
http://bugs.python.org/issue24124 opened by skip.montanaro
#24126: newlines attribute does not get set after calling readline()
http://bugs.python.org/issue24126 opened by arekfu
#24127: Fatal error in launcher: Job information querying failed
http://bugs.python.org/issue24127 opened by gavstar
#24129: Incorrect (misleading) statement in the execution model docume
http://bugs.python.org/issue24129 opened by levkivskyi
#24130: Remove -fno-common compile option from OS X framework builds?
http://bugs.python.org/issue24130 opened by ned.deily
#24131: [configparser] Add section/option delimiter to ExtendedInterpo
http://bugs.python.org/issue24131 opened by giflw
#24132: Direct sub-classing of pathlib.Path
http://bugs.python.org/issue24132 opened by projetmbc
#24136: document PEP 448
http://bugs.python.org/issue24136 opened by benjamin.peterson
#24137: Force not using _default_root in IDLE
http://bugs.python.org/issue24137 opened by serhiy.storchaka
#24138: Speed up range() by caching and modifying long objects
http://bugs.python.org/issue24138 opened by larry
#24139: Use sqlite3 extended error codes
http://bugs.python.org/issue24139 opened by Dima.Tisnek
#24140: In pdb using "until X" doesn't seem to have effect in commands
http://bugs.python.org/issue24140 opened by vyktor
#24142: ConfigParser._read doesn't join multi-line values collected wh
http://bugs.python.org/issue24142 opened by fhoech
#24143: Makefile in tarball don't provide make uninstall target
http://bugs.python.org/issue24143 opened by krichter
#24145: Support |= for parameters in converters
http://bugs.python.org/issue24145 opened by larry
Most recent 15 issues with no replies (15)
==========================================
#24143: Makefile in tarball don't provide make uninstall target
http://bugs.python.org/issue24143
#24140: In pdb using "until X" doesn't seem to have effect in commands
http://bugs.python.org/issue24140
#24137: Force not using _default_root in IDLE
http://bugs.python.org/issue24137
#24136: document PEP 448
http://bugs.python.org/issue24136
#24131: [configparser] Add section/option delimiter to ExtendedInterpo
http://bugs.python.org/issue24131
#24129: Incorrect (misleading) statement in the execution model docume
http://bugs.python.org/issue24129
#24115: PyObject_IsInstance() and PyObject_IsSubclass() can fail
http://bugs.python.org/issue24115
#24114: ctypes.utils uninitialized variable 'path'
http://bugs.python.org/issue24114
#24111: Valgrind suppression file should be updated
http://bugs.python.org/issue24111
#24104: Use after free in xmlparser_setevents (2)
http://bugs.python.org/issue24104
#24103: Use after free in xmlparser_setevents (1)
http://bugs.python.org/issue24103
#24097: Use after free in PyObject_GetState
http://bugs.python.org/issue24097
#24087: Documentation doesn't explain the term "coroutine" (PEP 342)
http://bugs.python.org/issue24087
#24084: pstats: sub-millisecond display
http://bugs.python.org/issue24084
#24063: Support Mageia and Arch Linux in the platform module
http://bugs.python.org/issue24063
Most recent 15 issues waiting for review (15)
=============================================
#24145: Support |= for parameters in converters
http://bugs.python.org/issue24145
#24142: ConfigParser._read doesn't join multi-line values collected wh
http://bugs.python.org/issue24142
#24138: Speed up range() by caching and modifying long objects
http://bugs.python.org/issue24138
#24130: Remove -fno-common compile option from OS X framework builds?
http://bugs.python.org/issue24130
#24117: Wrong range checking in GB18030 decoder.
http://bugs.python.org/issue24117
#24114: ctypes.utils uninitialized variable 'path'
http://bugs.python.org/issue24114
#24109: Documentation for difflib uses optparse
http://bugs.python.org/issue24109
#24102: Multiple type confusions in unicode error handlers
http://bugs.python.org/issue24102
#24091: Use after free in Element.extend (1)
http://bugs.python.org/issue24091
#24087: Documentation doesn't explain the term "coroutine" (PEP 342)
http://bugs.python.org/issue24087
#24084: pstats: sub-millisecond display
http://bugs.python.org/issue24084
#24082: Obsolete note in argument parsing (c-api/arg.rst)
http://bugs.python.org/issue24082
#24076: sum() several times slower on Python 3
http://bugs.python.org/issue24076
#24068: statistics module - incorrect results with boolean input
http://bugs.python.org/issue24068
#24064: Make the property doctstring writeable
http://bugs.python.org/issue24064
Top 10 most discussed issues (10)
=================================
#22906: PEP 479: Change StopIteration handling inside generators
http://bugs.python.org/issue22906 15 msgs
#20179: Derby #10: Convert 50 sites to Argument Clinic across 4 files
http://bugs.python.org/issue20179 13 msgs
#24132: Direct sub-classing of pathlib.Path
http://bugs.python.org/issue24132 11 msgs
#24127: Fatal error in launcher: Job information querying failed
http://bugs.python.org/issue24127 9 msgs
#22881: show median in benchmark results
http://bugs.python.org/issue22881 7 msgs
#24102: Multiple type confusions in unicode error handlers
http://bugs.python.org/issue24102 6 msgs
#21800: Implement RFC 6855 (IMAP Support for UTF-8) in imaplib.
http://bugs.python.org/issue21800 5 msgs
#23888: Fixing fractional expiry time bug in cookiejar
http://bugs.python.org/issue23888 5 msgs
#24085: large memory overhead when pyc is recompiled
http://bugs.python.org/issue24085 5 msgs
#24117: Wrong range checking in GB18030 decoder.
http://bugs.python.org/issue24117 5 msgs
Issues closed (43)
==================
#2292: Missing *-unpacking generalizations
http://bugs.python.org/issue2292 closed by benjamin.peterson
#20148: Derby: Convert the _sre module to use Argument Clinic
http://bugs.python.org/issue20148 closed by serhiy.storchaka
#20168: Derby: Convert the _tkinter module to use Argument Clinic
http://bugs.python.org/issue20168 closed by serhiy.storchaka
#20274: sqlite module has bad argument parsing code, including undefin
http://bugs.python.org/issue20274 closed by larry
#21520: Erroneous zipfile test failure if the string 'bad' appears in
http://bugs.python.org/issue21520 closed by larry
#22334: test_tcl.test_split() fails on "x86 FreeBSD 7.2 3.x" buildbot
http://bugs.python.org/issue22334 closed by serhiy.storchaka
#23330: h2py.py regular expression missing
http://bugs.python.org/issue23330 closed by serhiy.storchaka
#23880: Tkinter: getint and getdouble should support Tcl_Obj
http://bugs.python.org/issue23880 closed by serhiy.storchaka
#23911: Move path-based bootstrap code to a separate frozen file.
http://bugs.python.org/issue23911 closed by eric.snow
#23920: Should Clinic have "nullable" or types=NoneType?
http://bugs.python.org/issue23920 closed by larry
#24000: More fixes for the Clinic mapping of converters to format unit
http://bugs.python.org/issue24000 closed by larry
#24001: Clinic: use raw types in types= set
http://bugs.python.org/issue24001 closed by larry
#24051: Argument Clinic no longer works with single optional argument
http://bugs.python.org/issue24051 closed by serhiy.storchaka
#24060: Clearify necessities for logging with timestamps
http://bugs.python.org/issue24060 closed by python-dev
#24066: send_message should take all the addresses in the To: header i
http://bugs.python.org/issue24066 closed by kirelagin
#24081: Obsolete caveat in reload() docs
http://bugs.python.org/issue24081 closed by r.david.murray
#24088: yield expression confusion
http://bugs.python.org/issue24088 closed by gvanrossum
#24089: argparse crashes with AssertionError
http://bugs.python.org/issue24089 closed by ned.deily
#24092: Use after free in Element.extend (2)
http://bugs.python.org/issue24092 closed by serhiy.storchaka
#24093: Use after free in Element.remove
http://bugs.python.org/issue24093 closed by serhiy.storchaka
#24094: Use after free during json encoding (PyType_IsSubtype)
http://bugs.python.org/issue24094 closed by python-dev
#24095: Use after free during json encoding a dict (2)
http://bugs.python.org/issue24095 closed by benjamin.peterson
#24096: Use after free in get_filter
http://bugs.python.org/issue24096 closed by python-dev
#24099: Use after free in siftdown (1)
http://bugs.python.org/issue24099 closed by rhettinger
#24100: Use after free in siftdown (2)
http://bugs.python.org/issue24100 closed by rhettinger
#24101: Use after free in siftup
http://bugs.python.org/issue24101 closed by rhettinger
#24105: Use after free during json encoding a dict (3)
http://bugs.python.org/issue24105 closed by benjamin.peterson
#24106: Messed up indentation makes undesired piece of code being run!
http://bugs.python.org/issue24106 closed by r.david.murray
#24108: fnmatch.translate('*.txt') fails
http://bugs.python.org/issue24108 closed by r.david.murray
#24112: %b does not work, as a binary output formatter
http://bugs.python.org/issue24112 closed by steven.daprano
#24113: shlex constructor unreachable code
http://bugs.python.org/issue24113 closed by rhettinger
#24118: http.client example is no longer valid
http://bugs.python.org/issue24118 closed by python-dev
#24121: collections page doesn't mention that deques are mutable
http://bugs.python.org/issue24121 closed by rhettinger
#24122: Install fails after configure sets the extending/embedding ins
http://bugs.python.org/issue24122 closed by doko
#24123: Python 2.7 Tutorial Conflicting behavior with WeakValueDiction
http://bugs.python.org/issue24123 closed by jessembacon
#24125: Fix for #23865 breaks docutils
http://bugs.python.org/issue24125 closed by serhiy.storchaka
#24128: Documentation links are forwarded to Python 2
http://bugs.python.org/issue24128 closed by r.david.murray
#24133: Add 'composable' decorator to functools (with @ matrix multipl
http://bugs.python.org/issue24133 closed by r.david.murray
#24134: assertRaises can behave differently
http://bugs.python.org/issue24134 closed by serhiy.storchaka
#24135: Policy for altering sys.path
http://bugs.python.org/issue24135 closed by r.david.murray
#24141: Python 3 ships an outdated valgrind suppressison file.
http://bugs.python.org/issue24141 closed by ned.deily
#24144: Docs discourage use of binascii.unhexlify etc.
http://bugs.python.org/issue24144 closed by r.david.murray
#24146: ast.literal_eval doesn't support the Python ternary operator
http://bugs.python.org/issue24146 closed by r.david.murray
[View Less]
1
0
Hi python-dev,
Updated version of the PEP is below.
Quick summary of changes:
1. set_coroutine_wrapper and get_coroutine_wrapper functions
are now thread-specific (like settrace etc).
2. Updated Abstract & Rationale sections.
3. RuntimeWarning is always raised when a coroutine wasn't
awaited on. This is in addition to what 'set_coroutine_wrapper'
will/can do.
4. asyncio.async is renamed to asyncio.ensure_future; it will
be deprecated in 3.5.
5. Uses of async/await in CPython …
[View More]codebase are documented.
6. Other small edits and updates.
Thanks,
Yury
PEP: 492
Title: Coroutines with async and await syntax
Version: $Revision$
Last-Modified: $Date$
Author: Yury Selivanov <yselivanov(a)sprymix.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 09-Apr-2015
Python-Version: 3.5
Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015, 29-Apr-2015,
05-May-2015
Abstract
========
The growth of Internet and general connectivity has triggered the
proportionate need for responsive and scalable code. This proposal
aims to answer that need by making writing explicitly asynchronous,
concurrent Python code easier and more Pythonic.
It is proposed to make *coroutines* a proper standalone concept in
Python, and introduce new supporting syntax. The ultimate goal
is to help establish a common, easily approachable, mental
model of asynchronous programming in Python and make it as close to
synchronous programming as possible.
We believe that the changes proposed here will help keep Python
relevant and competitive in a quickly growing area of asynchronous
programming, as many other languages have adopted, or are planning to
adopt, similar features: [2]_, [5]_, [6]_, [7]_, [8]_, [10]_.
Rationale and Goals
===================
Current Python supports implementing coroutines via generators (PEP
342), further enhanced by the ``yield from`` syntax introduced in PEP
380. This approach has a number of shortcomings:
* It is easy to confuse coroutines with regular generators, since they
share the same syntax; this is especially true for new developers.
* Whether or not a function is a coroutine is determined by a presence
of ``yield`` or ``yield from`` statements in its *body*, which can
lead to unobvious errors when such statements appear in or disappear
from function body during refactoring.
* Support for asynchronous calls is limited to expressions where
``yield`` is allowed syntactically, limiting the usefulness of
syntactic features, such as ``with`` and ``for`` statements.
This proposal makes coroutines a native Python language feature, and
clearly separates them from generators. This removes
generator/coroutine ambiguity, and makes it possible to reliably define
coroutines without reliance on a specific library. This also enables
linters and IDEs to improve static code analysis and refactoring.
Native coroutines and the associated new syntax features make it
possible to define context manager and iteration protocols in
asynchronous terms. As shown later in this proposal, the new ``async
with`` statement lets Python programs perform asynchronous calls when
entering and exiting a runtime context, and the new ``async for``
statement makes it possible to perform asynchronous calls in iterators.
Specification
=============
This proposal introduces new syntax and semantics to enhance coroutine
support in Python.
This specification presumes knowledge of the implementation of
coroutines in Python (PEP 342 and PEP 380). Motivation for the syntax
changes proposed here comes from the asyncio framework (PEP 3156) and
the "Cofunctions" proposal (PEP 3152, now rejected in favor of this
specification).
From this point in this document we use the word *native coroutine* to
refer to functions declared using the new syntax. *generator-based
coroutine* is used where necessary to refer to coroutines that are
based on generator syntax. *coroutine* is used in contexts where both
definitions are applicable.
New Coroutine Declaration Syntax
--------------------------------
The following new syntax is used to declare a *native coroutine*::
async def read_data(db):
pass
Key properties of *coroutines*:
* ``async def`` functions are always coroutines, even if they do not
contain ``await`` expressions.
* It is a ``SyntaxError`` to have ``yield`` or ``yield from``
expressions in an ``async`` function.
* Internally, two new code object flags were introduced:
- ``CO_COROUTINE`` is used to enable runtime detection of
*coroutines* (and migrating existing code).
- ``CO_NATIVE_COROUTINE`` is used to mark *native coroutines*
(defined with new syntax.)
All coroutines have ``CO_COROUTINE``, ``CO_NATIVE_COROUTINE``, and
``CO_GENERATOR`` flags set.
* Regular generators, when called, return a *generator object*;
similarly, coroutines return a *coroutine object*.
* ``StopIteration`` exceptions are not propagated out of coroutines,
and are replaced with a ``RuntimeError``. For regular generators
such behavior requires a future import (see PEP 479).
* When a *coroutine* is garbage collected, a ``RuntimeWarning`` is
raised if it was never awaited on (see also `Debugging Features`_.)
* See also `Coroutine objects`_ section.
types.coroutine()
-----------------
A new function ``coroutine(gen)`` is added to the ``types`` module. It
allows interoperability between existing *generator-based coroutines*
in asyncio and *native coroutines* introduced by this PEP.
The function applies ``CO_COROUTINE`` flag to generator-function's code
object, making it return a *coroutine object*.
The function can be used as a decorator, since it modifies generator-
functions in-place and returns them.
Note, that the ``CO_NATIVE_COROUTINE`` flag is not applied by
``types.coroutine()`` to make it possible to separate *native
coroutines* defined with new syntax, from *generator-based coroutines*.
Await Expression
----------------
The following new ``await`` expression is used to obtain a result of
coroutine execution::
async def read_data(db):
data = await db.fetch('SELECT ...')
...
``await``, similarly to ``yield from``, suspends execution of
``read_data`` coroutine until ``db.fetch`` *awaitable* completes and
returns the result data.
It uses the ``yield from`` implementation with an extra step of
validating its argument. ``await`` only accepts an *awaitable*, which
can be one of:
* A *native coroutine object* returned from a *native coroutine*.
* A *generator-based coroutine object* returned from a generator
decorated with ``types.coroutine()``.
* An object with an ``__await__`` method returning an iterator.
Any ``yield from`` chain of calls ends with a ``yield``. This is a
fundamental mechanism of how *Futures* are implemented. Since,
internally, coroutines are a special kind of generators, every
``await`` is suspended by a ``yield`` somewhere down the chain of
``await`` calls (please refer to PEP 3156 for a detailed
explanation.)
To enable this behavior for coroutines, a new magic method called
``__await__`` is added. In asyncio, for instance, to enable *Future*
objects in ``await`` statements, the only change is to add
``__await__ = __iter__`` line to ``asyncio.Future`` class.
Objects with ``__await__`` method are called *Future-like* objects in
the rest of this PEP.
Also, please note that ``__aiter__`` method (see its definition
below) cannot be used for this purpose. It is a different protocol,
and would be like using ``__iter__`` instead of ``__call__`` for
regular callables.
It is a ``TypeError`` if ``__await__`` returns anything but an
iterator.
* Objects defined with CPython C API with a ``tp_await`` function,
returning an iterator (similar to ``__await__`` method).
It is a ``SyntaxError`` to use ``await`` outside of an ``async def``
function (like it is a ``SyntaxError`` to use ``yield`` outside of
``def`` function.)
It is a ``TypeError`` to pass anything other than an *awaitable* object
to an ``await`` expression.
Updated operator precedence table
'''''''''''''''''''''''''''''''''
``await`` keyword is defined as follows::
power ::= await ["**" u_expr]
await ::= ["await"] primary
where "primary" represents the most tightly bound operations of the
language. Its syntax is::
primary ::= atom | attributeref | subscription | slicing | call
See Python Documentation [12]_ and `Grammar Updates`_ section of this
proposal for details.
The key ``await`` difference from ``yield`` and ``yield from``
operators is that *await expressions* do not require parentheses around
them most of the times.
Also, ``yield from`` allows any expression as its argument, including
expressions like ``yield from a() + b()``, that would be parsed as
``yield from (a() + b())``, which is almost always a bug. In general,
the result of any arithmetic operation is not an *awaitable* object.
To avoid this kind of mistakes, it was decided to make ``await``
precedence lower than ``[]``, ``()``, and ``.``, but higher than ``**``
operators.
+------------------------------+-----------------------------------+
| Operator | Description |
+==============================+===================================+
| ``yield`` ``x``, | Yield expression |
| ``yield from`` ``x`` | |
+------------------------------+-----------------------------------+
| ``lambda`` | Lambda expression |
+------------------------------+-----------------------------------+
| ``if`` -- ``else`` | Conditional expression |
+------------------------------+-----------------------------------+
| ``or`` | Boolean OR |
+------------------------------+-----------------------------------+
| ``and`` | Boolean AND |
+------------------------------+-----------------------------------+
| ``not`` ``x`` | Boolean NOT |
+------------------------------+-----------------------------------+
| ``in``, ``not in``, | Comparisons, including membership |
| ``is``, ``is not``, ``<``, | tests and identity tests |
| ``<=``, ``>``, ``>=``, | |
| ``!=``, ``==`` | |
+------------------------------+-----------------------------------+
| ``|`` | Bitwise OR |
+------------------------------+-----------------------------------+
| ``^`` | Bitwise XOR |
+------------------------------+-----------------------------------+
| ``&`` | Bitwise AND |
+------------------------------+-----------------------------------+
| ``<<``, ``>>`` | Shifts |
+------------------------------+-----------------------------------+
| ``+``, ``-`` | Addition and subtraction |
+------------------------------+-----------------------------------+
| ``*``, ``@``, ``/``, ``//``, | Multiplication, matrix |
| ``%`` | multiplication, division, |
| | remainder |
+------------------------------+-----------------------------------+
| ``+x``, ``-x``, ``~x`` | Positive, negative, bitwise NOT |
+------------------------------+-----------------------------------+
| ``**`` | Exponentiation |
+------------------------------+-----------------------------------+
| ``await`` ``x`` | Await expression |
+------------------------------+-----------------------------------+
| ``x[index]``, | Subscription, slicing, |
| ``x[index:index]``, | call, attribute reference |
| ``x(arguments...)``, | |
| ``x.attribute`` | |
+------------------------------+-----------------------------------+
| ``(expressions...)``, | Binding or tuple display, |
| ``[expressions...]``, | list display, |
| ``{key: value...}``, | dictionary display, |
| ``{expressions...}`` | set display |
+------------------------------+-----------------------------------+
Examples of "await" expressions
'''''''''''''''''''''''''''''''
Valid syntax examples:
================================== ==================================
Expression Will be parsed as
================================== ==================================
``if await fut: pass`` ``if (await fut): pass``
``if await fut + 1: pass`` ``if (await fut) + 1: pass``
``pair = await fut, 'spam'`` ``pair = (await fut), 'spam'``
``with await fut, open(): pass`` ``with (await fut), open(): pass``
``await foo()['spam'].baz()()`` ``await ( foo()['spam'].baz()() )``
``return await coro()`` ``return ( await coro() )``
``res = await coro() ** 2`` ``res = (await coro()) ** 2``
``func(a1=await coro(), a2=0)`` ``func(a1=(await coro()), a2=0)``
``await foo() + await bar()`` ``(await foo()) + (await bar())``
``-await foo()`` ``-(await foo())``
================================== ==================================
Invalid syntax examples:
================================== ==================================
Expression Should be written as
================================== ==================================
``await await coro()`` ``await (await coro())``
``await -coro()`` ``await (-coro())``
================================== ==================================
Asynchronous Context Managers and "async with"
----------------------------------------------
An *asynchronous context manager* is a context manager that is able to
suspend execution in its *enter* and *exit* methods.
To make this possible, a new protocol for asynchronous context managers
is proposed. Two new magic methods are added: ``__aenter__`` and
``__aexit__``. Both must return an *awaitable*.
An example of an asynchronous context manager::
class AsyncContextManager:
async def __aenter__(self):
await log('entering context')
async def __aexit__(self, exc_type, exc, tb):
await log('exiting context')
New Syntax
''''''''''
A new statement for asynchronous context managers is proposed::
async with EXPR as VAR:
BLOCK
which is semantically equivalent to::
mgr = (EXPR)
aexit = type(mgr).__aexit__
aenter = type(mgr).__aenter__(mgr)
exc = True
try:
VAR = await aenter
BLOCK
except:
if not await aexit(mgr, *sys.exc_info()):
raise
else:
await aexit(mgr, None, None, None)
As with regular ``with`` statements, it is possible to specify multiple
context managers in a single ``async with`` statement.
It is an error to pass a regular context manager without ``__aenter__``
and ``__aexit__`` methods to ``async with``. It is a ``SyntaxError``
to use ``async with`` outside of an ``async def`` function.
Example
'''''''
With *asynchronous context managers* it is easy to implement proper
database transaction managers for coroutines::
async def commit(session, data):
...
async with session.transaction():
...
await session.update(data)
...
Code that needs locking also looks lighter::
async with lock:
...
instead of::
with (yield from lock):
...
Asynchronous Iterators and "async for"
--------------------------------------
An *asynchronous iterable* is able to call asynchronous code in its
*iter* implementation, and *asynchronous iterator* can call
asynchronous code in its *next* method. To support asynchronous
iteration:
1. An object must implement an ``__aiter__`` method returning an
*awaitable* resulting in an *asynchronous iterator object*.
2. An *asynchronous iterator object* must implement an ``__anext__``
method returning an *awaitable*.
3. To stop iteration ``__anext__`` must raise a ``StopAsyncIteration``
exception.
An example of asynchronous iterable::
class AsyncIterable:
async def __aiter__(self):
return self
async def __anext__(self):
data = await self.fetch_data()
if data:
return data
else:
raise StopAsyncIteration
async def fetch_data(self):
...
New Syntax
''''''''''
A new statement for iterating through asynchronous iterators is
proposed::
async for TARGET in ITER:
BLOCK
else:
BLOCK2
which is semantically equivalent to::
iter = (ITER)
iter = await type(iter).__aiter__(iter)
running = True
while running:
try:
TARGET = await type(iter).__anext__(iter)
except StopAsyncIteration:
running = False
else:
BLOCK
else:
BLOCK2
It is a ``TypeError`` to pass a regular iterable without ``__aiter__``
method to ``async for``. It is a ``SyntaxError`` to use ``async for``
outside of an ``async def`` function.
As for with regular ``for`` statement, ``async for`` has an optional
``else`` clause.
Example 1
'''''''''
With asynchronous iteration protocol it is possible to asynchronously
buffer data during iteration::
async for data in cursor:
...
Where ``cursor`` is an asynchronous iterator that prefetches ``N`` rows
of data from a database after every ``N`` iterations.
The following code illustrates new asynchronous iteration protocol::
class Cursor:
def __init__(self):
self.buffer = collections.deque()
def _prefetch(self):
...
async def __aiter__(self):
return self
async def __anext__(self):
if not self.buffer:
self.buffer = await self._prefetch()
if not self.buffer:
raise StopAsyncIteration
return self.buffer.popleft()
then the ``Cursor`` class can be used as follows::
async for row in Cursor():
print(row)
which would be equivalent to the following code::
i = await Cursor().__aiter__()
while True:
try:
row = await i.__anext__()
except StopAsyncIteration:
break
else:
print(row)
Example 2
'''''''''
The following is a utility class that transforms a regular iterable to
an asynchronous one. While this is not a very useful thing to do, the
code illustrates the relationship between regular and asynchronous
iterators.
::
class AsyncIteratorWrapper:
def __init__(self, obj):
self._it = iter(obj)
async def __aiter__(self):
return self
async def __anext__(self):
try:
value = next(self._it)
except StopIteration:
raise StopAsyncIteration
return value
async for letter in AsyncIteratorWrapper("abc"):
print(letter)
Why StopAsyncIteration?
'''''''''''''''''''''''
Coroutines are still based on generators internally. So, before PEP
479, there was no fundamental difference between
::
def g1():
yield from fut
return 'spam'
and
::
def g2():
yield from fut
raise StopIteration('spam')
And since PEP 479 is accepted and enabled by default for coroutines,
the following example will have its ``StopIteration`` wrapped into a
``RuntimeError``
::
async def a1():
await fut
raise StopIteration('spam')
The only way to tell the outside code that the iteration has ended is
to raise something other than ``StopIteration``. Therefore, a new
built-in exception class ``StopAsyncIteration`` was added.
Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions
raised in coroutines are wrapped in ``RuntimeError``.
Coroutine objects
-----------------
Differences from generators
'''''''''''''''''''''''''''
This section applies only to *native coroutines* with
``CO_NATIVE_COROUTINE`` flag, i.e. defined with the new ``async def``
syntax.
**The behavior of existing *generator-based coroutines* in asyncio
remains unchanged.**
Great effort has been made to make sure that coroutines and
generators are treated as distinct concepts:
1. *Native coroutine objects* do not implement ``__iter__`` and
``__next__`` methods. Therefore, they cannot be iterated over or
passed to ``iter()``, ``list()``, ``tuple()`` and other built-ins.
They also cannot be used in a ``for..in`` loop.
An attempt to use ``__iter__`` or ``__next__`` on a *native
coroutine object* will result in a ``TypeError``.
2. *Plain generators* cannot ``yield from`` *native coroutine objects*:
doing so will result in a ``TypeError``.
3. *generator-based coroutines* (for asyncio code must be decorated
with ``(a)asyncio.coroutine``) can ``yield from`` *native coroutine
objects*.
4. ``inspect.isgenerator()`` and ``inspect.isgeneratorfunction()``
return ``False`` for *native coroutine objects* and *native
coroutine functions*.
Coroutine object methods
''''''''''''''''''''''''
Coroutines are based on generators internally, thus they share the
implementation. Similarly to generator objects, coroutine objects have
``throw()``, ``send()`` and ``close()`` methods. ``StopIteration`` and
``GeneratorExit`` play the same role for coroutine objects (although
PEP 479 is enabled by default for coroutines). See PEP 342, PEP 380,
and Python Documentation [11]_ for details.
``throw()``, ``send()`` methods for coroutine objects are used to push
values and raise errors into *Future-like* objects.
Debugging Features
------------------
A common beginner mistake is forgetting to use ``yield from`` on
coroutines::
@asyncio.coroutine
def useful():
asyncio.sleep(1) # this will do noting without 'yield from'
For debugging this kind of mistakes there is a special debug mode in
asyncio, in which ``@coroutine`` decorator wraps all functions with a
special object with a destructor logging a warning. Whenever a wrapped
generator gets garbage collected, a detailed logging message is
generated with information about where exactly the decorator function
was defined, stack trace of where it was collected, etc. Wrapper
object also provides a convenient ``__repr__`` function with detailed
information about the generator.
The only problem is how to enable these debug capabilities. Since
debug facilities should be a no-op in production mode, ``@coroutine``
decorator makes the decision of whether to wrap or not to wrap based on
an OS environment variable ``PYTHONASYNCIODEBUG``. This way it is
possible to run asyncio programs with asyncio's own functions
instrumented. ``EventLoop.set_debug``, a different debug facility, has
no impact on ``@coroutine`` decorator's behavior.
With this proposal, coroutines is a native, distinct from generators,
concept. *In addition* to a ``RuntimeWarning`` being raised on
coroutines that were never awaited, it is proposed to add two new
functions to the ``sys`` module: ``set_coroutine_wrapper`` and
``get_coroutine_wrapper``. This is to enable advanced debugging
facilities in asyncio and other frameworks (such as displaying where
exactly coroutine was created, and a more detailed stack trace of where
it was garbage collected).
New Standard Library Functions
------------------------------
* ``types.coroutine(gen)``. See `types.coroutine()`_ section for
details.
* ``inspect.iscoroutine(obj)`` returns ``True`` if ``obj`` is a
*coroutine object*.
* ``inspect.iscoroutinefunction(obj)`` returns ``True`` if ``obj`` is a
*coroutine function*.
* ``inspect.isawaitable(obj)`` returns ``True`` if ``obj`` can be used
in ``await`` expression. See `Await Expression`_ for details.
* ``sys.set_coroutine_wrapper(wrapper)`` allows to intercept creation
of *coroutine objects*. ``wrapper`` must be a callable that accepts
one argument: a *coroutine object* or ``None``. ``None`` resets the
wrapper. If called twice, the new wrapper replaces the previous one.
The function is thread-specific. See `Debugging Features`_ for more
details.
* ``sys.get_coroutine_wrapper()`` returns the current wrapper object.
Returns ``None`` if no wrapper was set. The function is
thread-specific. See `Debugging Features`_ for more details.
Glossary
========
:Native coroutine:
A coroutine function is declared with ``async def``. It uses
``await`` and ``return value``; see `New Coroutine Declaration
Syntax`_ for details.
:Native coroutine object:
Returned from a native coroutine function. See `Await Expression`_
for details.
:Generator-based coroutine:
Coroutines based on generator syntax. Most common example are
functions decorated with ``(a)asyncio.coroutine``.
:Generator-based coroutine object:
Returned from a generator-based coroutine function.
:Coroutine:
Either *native coroutine* or *generator-based coroutine*.
:Coroutine object:
Either *native coroutine object* or *generator-based coroutine
object*.
:Future-like object:
An object with an ``__await__`` method, or a C object with
``tp_await`` function, returning an iterator. Can be consumed by
an ``await`` expression in a coroutine. A coroutine waiting for a
Future-like object is suspended until the Future-like object's
``__await__`` completes, and returns the result. See `Await
Expression`_ for details.
:Awaitable:
A *Future-like* object or a *coroutine object*. See `Await
Expression`_ for details.
:Asynchronous context manager:
An asynchronous context manager has ``__aenter__`` and ``__aexit__``
methods and can be used with ``async with``. See `Asynchronous
Context Managers and "async with"`_ for details.
:Asynchronous iterable:
An object with an ``__aiter__`` method, which must return an
*asynchronous iterator* object. Can be used with ``async for``.
See `Asynchronous Iterators and "async for"`_ for details.
:Asynchronous iterator:
An asynchronous iterator has an ``__anext__`` method. See
`Asynchronous Iterators and "async for"`_ for details.
List of functions and methods
=============================
================= =================================== =================
Method Can contain Can't contain
================= =================================== =================
async def func await, return value yield, yield from
async def __a*__ await, return value yield, yield from
def __a*__ return awaitable await
def __await__ yield, yield from, return iterable await
generator yield, yield from, return value await
================= =================================== =================
Where:
* "async def func": native coroutine;
* "async def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``,
``__aexit__`` defined with the ``async`` keyword;
* "def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``,
``__aexit__`` defined without the ``async`` keyword, must return an
*awaitable*;
* "def __await__": ``__await__`` method to implement *Future-like*
objects;
* generator: a "regular" generator, function defined with ``def`` and
which contains a least one ``yield`` or ``yield from`` expression.
Transition Plan
===============
To avoid backwards compatibility issues with ``async`` and ``await``
keywords, it was decided to modify ``tokenizer.c`` in such a way, that
it:
* recognizes ``async def`` ``NAME`` tokens combination;
* keeps track of regular ``def`` and ``async def`` indented blocks;
* while tokenizing ``async def`` block, it replaces ``'async'``
``NAME`` token with ``ASYNC``, and ``'await'`` ``NAME`` token with
``AWAIT``;
* while tokenizing ``def`` block, it yields ``'async'`` and ``'await'``
``NAME`` tokens as is.
This approach allows for seamless combination of new syntax features
(all of them available only in ``async`` functions) with any existing
code.
An example of having "async def" and "async" attribute in one piece of
code::
class Spam:
async = 42
async def ham():
print(getattr(Spam, 'async'))
# The coroutine can be executed and will print '42'
Backwards Compatibility
-----------------------
This proposal preserves 100% backwards compatibility.
asyncio
'''''''
``asyncio`` module was adapted and tested to work with coroutines and
new statements. Backwards compatibility is 100% preserved, i.e. all
existing code will work as-is.
The required changes are mainly:
1. Modify ``(a)asyncio.coroutine`` decorator to use new
``types.coroutine()`` function.
2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class.
3. Add ``ensure_future()`` as an alias for ``async()`` function.
Deprecate ``async()`` function.
asyncio migration strategy
''''''''''''''''''''''''''
Because *plain generators* cannot ``yield from`` *native coroutine
objects* (see `Differences from generators`_ section for more details),
it is advised to make sure that all generator-based coroutines are
decorated with ``(a)asyncio.coroutine`` *before* starting to use the new
syntax.
async/await in CPython code base
''''''''''''''''''''''''''''''''
There is no use of ``await`` names in CPython.
``async`` is mostly used by asyncio. We are addressing this by
renaming ``async()`` function to ``ensure_future()`` (see `asyncio`_
section for details.)
Another use of ``async`` keyword is in ``Lib/xml/dom/xmlbuilder.py``,
to define an ``async = False`` attribute for ``DocumentLS`` class.
There is no documentation or tests for it, it is not used anywhere else
in CPython. It is replaced with a getter, that raises a
``DeprecationWarning``, advising to use ``async_`` attribute instead.
'async' attribute is not documented and is not used in CPython code
base.
Grammar Updates
---------------
Grammar changes are fairly minimal::
decorated: decorators (classdef | funcdef | async_funcdef)
async_funcdef: ASYNC funcdef
compound_stmt: (if_stmt | while_stmt | for_stmt | try_stmt | with_stmt
| funcdef | classdef | decorated | async_stmt)
async_stmt: ASYNC (funcdef | with_stmt | for_stmt)
power: atom_expr ['**' factor]
atom_expr: [AWAIT] atom trailer*
Transition Period Shortcomings
------------------------------
There is just one.
Until ``async`` and ``await`` are not proper keywords, it is not
possible (or at least very hard) to fix ``tokenizer.c`` to recognize
them on the **same line** with ``def`` keyword::
# async and await will always be parsed as variables
async def outer(): # 1
def nested(a=(await fut)):
pass
async def foo(): return (await fut) # 2
Since ``await`` and ``async`` in such cases are parsed as ``NAME``
tokens, a ``SyntaxError`` will be raised.
To workaround these issues, the above examples can be easily rewritten
to a more readable form::
async def outer(): # 1
a_default = await fut
def nested(a=a_default):
pass
async def foo(): # 2
return (await fut)
This limitation will go away as soon as ``async`` and ``await`` are
proper keywords.
Deprecation Plans
-----------------
``async`` and ``await`` names will be softly deprecated in CPython 3.5
and 3.6. In 3.7 we will transform them to proper keywords. Making
``async`` and ``await`` proper keywords before 3.7 might make it harder
for people to port their code to Python 3.
Design Considerations
=====================
PEP 3152
--------
PEP 3152 by Gregory Ewing proposes a different mechanism for coroutines
(called "cofunctions"). Some key points:
1. A new keyword ``codef`` to declare a *cofunction*. *Cofunction* is
always a generator, even if there is no ``cocall`` expressions
inside it. Maps to ``async def`` in this proposal.
2. A new keyword ``cocall`` to call a *cofunction*. Can only be used
inside a *cofunction*. Maps to ``await`` in this proposal (with
some differences, see below.)
3. It is not possible to call a *cofunction* without a ``cocall``
keyword.
4. ``cocall`` grammatically requires parentheses after it::
atom: cocall | <existing alternatives for atom>
cocall: 'cocall' atom cotrailer* '(' [arglist] ')'
cotrailer: '[' subscriptlist ']' | '.' NAME
5. ``cocall f(*args, **kwds)`` is semantically equivalent to
``yield from f.__cocall__(*args, **kwds)``.
Differences from this proposal:
1. There is no equivalent of ``__cocall__`` in this PEP, which is
called and its result is passed to ``yield from`` in the ``cocall``
expression. ``await`` keyword expects an *awaitable* object,
validates the type, and executes ``yield from`` on it. Although,
``__await__`` method is similar to ``__cocall__``, but is only used
to define *Future-like* objects.
2. ``await`` is defined in almost the same way as ``yield from`` in the
grammar (it is later enforced that ``await`` can only be inside
``async def``). It is possible to simply write ``await future``,
whereas ``cocall`` always requires parentheses.
3. To make asyncio work with PEP 3152 it would be required to modify
``(a)asyncio.coroutine`` decorator to wrap all functions in an object
with a ``__cocall__`` method, or to implement ``__cocall__`` on
generators. To call *cofunctions* from existing generator-based
coroutines it would be required to use ``costart(cofunc, *args,
**kwargs)`` built-in.
4. Since it is impossible to call a *cofunction* without a ``cocall``
keyword, it automatically prevents the common mistake of forgetting
to use ``yield from`` on generator-based coroutines. This proposal
addresses this problem with a different approach, see `Debugging
Features`_.
5. A shortcoming of requiring a ``cocall`` keyword to call a coroutine
is that if is decided to implement coroutine-generators --
coroutines with ``yield`` or ``async yield`` expressions -- we
wouldn't need a ``cocall`` keyword to call them. So we'll end up
having ``__cocall__`` and no ``__call__`` for regular coroutines,
and having ``__call__`` and no ``__cocall__`` for coroutine-
generators.
6. Requiring parentheses grammatically also introduces a whole lot
of new problems.
The following code::
await fut
await function_returning_future()
await asyncio.gather(coro1(arg1, arg2), coro2(arg1, arg2))
would look like::
cocall fut() # or cocall costart(fut)
cocall (function_returning_future())()
cocall asyncio.gather(costart(coro1, arg1, arg2),
costart(coro2, arg1, arg2))
7. There are no equivalents of ``async for`` and ``async with`` in PEP
3152.
Coroutine-generators
--------------------
With ``async for`` keyword it is desirable to have a concept of a
*coroutine-generator* -- a coroutine with ``yield`` and ``yield from``
expressions. To avoid any ambiguity with regular generators, we would
likely require to have an ``async`` keyword before ``yield``, and
``async yield from`` would raise a ``StopAsyncIteration`` exception.
While it is possible to implement coroutine-generators, we believe that
they are out of scope of this proposal. It is an advanced concept that
should be carefully considered and balanced, with a non-trivial changes
in the implementation of current generator objects. This is a matter
for a separate PEP.
Why "async" and "await" keywords
--------------------------------
async/await is not a new concept in programming languages:
* C# has it since long time ago [5]_;
* proposal to add async/await in ECMAScript 7 [2]_;
see also Traceur project [9]_;
* Facebook's Hack/HHVM [6]_;
* Google's Dart language [7]_;
* Scala [8]_;
* proposal to add async/await to C++ [10]_;
* and many other less popular languages.
This is a huge benefit, as some users already have experience with
async/await, and because it makes working with many languages in one
project easier (Python with ECMAScript 7 for instance).
Why "__aiter__" returns awaitable
---------------------------------
In principle, ``__aiter__`` could be a regular function. There are
several good reasons to make it a coroutine:
* as most of the ``__anext__``, ``__aenter__``, and ``__aexit__``
methods are coroutines, users would often make a mistake defining it
as ``async`` anyways;
* there might be a need to run some asynchronous operations in
``__aiter__``, for instance to prepare DB queries or do some file
operation.
Importance of "async" keyword
-----------------------------
While it is possible to just implement ``await`` expression and treat
all functions with at least one ``await`` as coroutines, this approach
makes APIs design, code refactoring and its long time support harder.
Let's pretend that Python only has ``await`` keyword::
def useful():
...
await log(...)
...
def important():
await useful()
If ``useful()`` function is refactored and someone removes all
``await`` expressions from it, it would become a regular python
function, and all code that depends on it, including ``important()``
would be broken. To mitigate this issue a decorator similar to
``(a)asyncio.coroutine`` has to be introduced.
Why "async def"
---------------
For some people bare ``async name(): pass`` syntax might look more
appealing than ``async def name(): pass``. It is certainly easier to
type. But on the other hand, it breaks the symmetry between ``async
def``, ``async with`` and ``async for``, where ``async`` is a modifier,
stating that the statement is asynchronous. It is also more consistent
with the existing grammar.
Why not "await for" and "await with"
------------------------------------
``async`` is an adjective, and hence it is a better choice for a
*statement qualifier* keyword. ``await for/with`` would imply that
something is awaiting for a completion of a ``for`` or ``with``
statement.
Why "async def" and not "def async"
-----------------------------------
``async`` keyword is a *statement qualifier*. A good analogy to it are
"static", "public", "unsafe" keywords from other languages. "async
for" is an asynchronous "for" statement, "async with" is an
asynchronous "with" statement, "async def" is an asynchronous function.
Having "async" after the main statement keyword might introduce some
confusion, like "for async item in iterator" can be read as "for each
asynchronous item in iterator".
Having ``async`` keyword before ``def``, ``with`` and ``for`` also
makes the language grammar simpler. And "async def" better separates
coroutines from regular functions visually.
Why not a __future__ import
---------------------------
`Transition Plan`_ section explains how tokenizer is modified to treat
``async`` and ``await`` as keywords *only* in ``async def`` blocks.
Hence ``async def`` fills the role that a module level compiler
declaration like ``from __future__ import async_await`` would otherwise
fill.
Why magic methods start with "a"
--------------------------------
New asynchronous magic methods ``__aiter__``, ``__anext__``,
``__aenter__``, and ``__aexit__`` all start with the same prefix "a".
An alternative proposal is to use "async" prefix, so that ``__aiter__``
becomes ``__async_iter__``. However, to align new magic methods with
the existing ones, such as ``__radd__`` and ``__iadd__`` it was decided
to use a shorter version.
Why not reuse existing magic names
----------------------------------
An alternative idea about new asynchronous iterators and context
managers was to reuse existing magic methods, by adding an ``async``
keyword to their declarations::
class CM:
async def __enter__(self): # instead of __aenter__
...
This approach has the following downsides:
* it would not be possible to create an object that works in both
``with`` and ``async with`` statements;
* it would break backwards compatibility, as nothing prohibits from
returning a Future-like objects from ``__enter__`` and/or
``__exit__`` in Python <= 3.4;
* one of the main points of this proposal is to make native coroutines
as simple and foolproof as possible, hence the clear separation of
the protocols.
Why not reuse existing "for" and "with" statements
--------------------------------------------------
The vision behind existing generator-based coroutines and this proposal
is to make it easy for users to see where the code might be suspended.
Making existing "for" and "with" statements to recognize asynchronous
iterators and context managers will inevitably create implicit suspend
points, making it harder to reason about the code.
Comprehensions
--------------
Syntax for asynchronous comprehensions could be provided, but this
construct is outside of the scope of this PEP.
Async lambda functions
----------------------
Syntax for asynchronous lambda functions could be provided, but this
construct is outside of the scope of this PEP.
Performance
===========
Overall Impact
--------------
This proposal introduces no observable performance impact. Here is an
output of python's official set of benchmarks [4]_:
::
python perf.py -r -b default ../cpython/python.exe
../cpython-aw/python.exe
[skipped]
Report on Darwin ysmac 14.3.0 Darwin Kernel Version 14.3.0:
Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64
x86_64 i386
Total CPU cores: 8
### etree_iterparse ###
Min: 0.365359 -> 0.349168: 1.05x faster
Avg: 0.396924 -> 0.379735: 1.05x faster
Significant (t=9.71)
Stddev: 0.01225 -> 0.01277: 1.0423x larger
The following not significant results are hidden, use -v to show them:
django_v2, 2to3, etree_generate, etree_parse, etree_process,
fastpickle,
fastunpickle, json_dump_v2, json_load, nbody, regex_v8, tornado_http.
Tokenizer modifications
-----------------------
There is no observable slowdown of parsing python files with the
modified tokenizer: parsing of one 12Mb file
(``Lib/test/test_binop.py`` repeated 1000 times) takes the same amount
of time.
async/await
-----------
The following micro-benchmark was used to determine performance
difference between "async" functions and generators::
import sys
import time
def binary(n):
if n <= 0:
return 1
l = yield from binary(n - 1)
r = yield from binary(n - 1)
return l + 1 + r
async def abinary(n):
if n <= 0:
return 1
l = await abinary(n - 1)
r = await abinary(n - 1)
return l + 1 + r
def timeit(gen, depth, repeat):
t0 = time.time()
for _ in range(repeat):
list(gen(depth))
t1 = time.time()
print('{}({}) * {}: total {:.3f}s'.format(
gen.__name__, depth, repeat, t1-t0))
The result is that there is no observable performance difference.
Minimum timing of 3 runs
::
abinary(19) * 30: total 12.985s
binary(19) * 30: total 12.953s
Note that depth of 19 means 1,048,575 calls.
Reference Implementation
========================
The reference implementation can be found here: [3]_.
List of high-level changes and new protocols
--------------------------------------------
1. New syntax for defining coroutines: ``async def`` and new ``await``
keyword.
2. New ``__await__`` method for Future-like objects, and new
``tp_await`` slot in ``PyTypeObject``.
3. New syntax for asynchronous context managers: ``async with``. And
associated protocol with ``__aenter__`` and ``__aexit__`` methods.
4. New syntax for asynchronous iteration: ``async for``. And
associated protocol with ``__aiter__``, ``__aexit__`` and new built-
in exception ``StopAsyncIteration``.
5. New AST nodes: ``AsyncFunctionDef``, ``AsyncFor``, ``AsyncWith``,
``Await``.
6. New functions: ``sys.set_coroutine_wrapper(callback)``,
``sys.get_coroutine_wrapper()``, ``types.coroutine(gen)``,
``inspect.iscoroutinefunction()``, ``inspect.iscoroutine()``,
and ``inspect.isawaitable()``.
7. New ``CO_COROUTINE`` and ``CO_NATIVE_COROUTINE`` bit flags for code
objects.
While the list of changes and new things is not short, it is important
to understand, that most users will not use these features directly.
It is intended to be used in frameworks and libraries to provide users
with convenient to use and unambiguous APIs with ``async def``,
``await``, ``async for`` and ``async with`` syntax.
Working example
---------------
All concepts proposed in this PEP are implemented [3]_ and can be
tested.
::
import asyncio
async def echo_server():
print('Serving on localhost:8000')
await asyncio.start_server(handle_connection,
'localhost', 8000)
async def handle_connection(reader, writer):
print('New connection...')
while True:
data = await reader.read(8192)
if not data:
break
print('Sending {:.10}... back'.format(repr(data)))
writer.write(data)
loop = asyncio.get_event_loop()
loop.run_until_complete(echo_server())
try:
loop.run_forever()
finally:
loop.close()
References
==========
.. [1] https://docs.python.org/3/library/asyncio-task.html#asyncio.coroutine
.. [2] http://wiki.ecmascript.org/doku.php?id=strawman:async_functions
.. [3] https://github.com/1st1/cpython/tree/await
.. [4] https://hg.python.org/benchmarks
.. [5] https://msdn.microsoft.com/en-us/library/hh191443.aspx
.. [6] http://docs.hhvm.com/manual/en/hack.async.php
.. [7] https://www.dartlang.org/articles/await-async/
.. [8] http://docs.scala-lang.org/sips/pending/async.html
.. [9]
https://github.com/google/traceur-compiler/wiki/LanguageFeatures#async-func…
.. [10]
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf (PDF)
.. [11]
https://docs.python.org/3/reference/expressions.html#generator-iterator-met…
.. [12] https://docs.python.org/3/reference/expressions.html#primaries
Acknowledgments
===============
I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Andrew
Svetlov, and Łukasz Langa for their initial feedback.
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
[View Less]
10
27
>From the PEP:
> Why not a __future__ import
>
> __future__ imports are inconvenient and easy to forget to add.
That is a horrible rationale for not using an import. By that logic we
should have everything in built-ins. ;)
> Working example
> ...
The working example only uses async def and await, not async with
nor async for nor __aenter__, etc., etc.
Could you put in a more complete example -- maybe a basic chat room
with both server and client -- that demonstrated …
[View More]more of the new
possibilities?
Having gone through the PEP again, I am still no closer to understanding
what happens here:
data = await reader.read(8192)
What does the flow of control look like at the interpreter level?
--
~Ethan~
[View Less]
12
27
Hi python-dev,
New version of the PEP is attached. Summary of updates:
1. Terminology:
- *native coroutine* term is used for "async def" functions.
- *generator-based coroutine* term is used for PEP 380
coroutines used in asyncio.
- *coroutine* is used when *native coroutine* or
*generator based coroutine* can be used in the same
context.
I think that it's not really productive to discuss the
terminology that we use in the PEP. Its only purpose is
to disambiguate concepts used in the …
[View More]PEP. We should discuss
how we will name new 'async def' coroutines in Python
Documentation if the PEP is accepted. Although if you
notice that somewhere in the PEP it is not crystal clear
what "coroutine" means please give me a heads up!
2. Syntax of await expressions is now thoroghly defined
in the PEP. See "Await Expression", "Updated operator
precedence table", and "Examples of "await" expressions"
sections.
I like the current approach. I'm still not convinced
that we should make 'await' the same grammatically as
unary minus.
I don't understand why we should allow parsing things
like 'await -fut'; this should be a SyntaxError.
I'm fine to modify the grammar to allow 'await await fut'
syntax, though. And I'm open to discussion :)
3. CO_NATIVE_COROUTINE flag. This enables us to disable
__iter__ and __next__ on native coroutines while maintaining
full backwards compatibility.
4. asyncio / Migration strategy. Existing code can
be used with PEP 492 as is, everything will work as
expected.
However, since *plain generator*s (not decorated with
@asyncio.coroutine) cannot 'yield from' native coroutines
(async def functions), it might break some code
*while adapting it to the new syntax*.
I'm open to just throw a RuntimeWarning in this case
in 3.5, and raise a TypeError in 3.6.
Please see the "Differences from generators" section of
the PEP.
5. inspect.isawaitable() function. And, all new functions
are now listed in the "New Standard Library Functions"
section.
6. Lot's of small updates and tweaks throughout the PEP.
Thanks,
Yury
PEP: 492
Title: Coroutines with async and await syntax
Version: $Revision$
Last-Modified: $Date$
Author: Yury Selivanov <yselivanov(a)sprymix.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 09-Apr-2015
Python-Version: 3.5
Post-History: 17-Apr-2015, 21-Apr-2015, 27-Apr-2015, 29-Apr-2015
Abstract
========
This PEP introduces new syntax for coroutines, asynchronous ``with``
statements and ``for`` loops. The main motivation behind this proposal
is to streamline writing and maintaining asynchronous code, as well as
to simplify previously hard to implement code patterns.
Rationale and Goals
===================
Current Python supports implementing coroutines via generators (PEP
342), further enhanced by the ``yield from`` syntax introduced in PEP
380. This approach has a number of shortcomings:
* it is easy to confuse coroutines with regular generators, since they
share the same syntax; async libraries often attempt to alleviate
this by using decorators (e.g. ``(a)asyncio.coroutine`` [1]_);
* it is not possible to natively define a coroutine which has no
``yield`` or ``yield from`` statements, again requiring the use of
decorators to fix potential refactoring issues;
* support for asynchronous calls is limited to expressions where
``yield`` is allowed syntactically, limiting the usefulness of
syntactic features, such as ``with`` and ``for`` statements.
This proposal makes coroutines a native Python language feature, and
clearly separates them from generators. This removes
generator/coroutine ambiguity, and makes it possible to reliably define
coroutines without reliance on a specific library. This also enables
linters and IDEs to improve static code analysis and refactoring.
Native coroutines and the associated new syntax features make it
possible to define context manager and iteration protocols in
asynchronous terms. As shown later in this proposal, the new ``async
with`` statement lets Python programs perform asynchronous calls when
entering and exiting a runtime context, and the new ``async for``
statement makes it possible to perform asynchronous calls in iterators.
Specification
=============
This proposal introduces new syntax and semantics to enhance coroutine
support in Python.
This specification presumes knowledge of the implementation of
coroutines in Python (PEP 342 and PEP 380). Motivation for the syntax
changes proposed here comes from the asyncio framework (PEP 3156) and
the "Cofunctions" proposal (PEP 3152, now rejected in favor of this
specification).
From this point in this document we use the word *native coroutine* to
refer to functions declared using the new syntax. *generator-based
coroutine* is used where necessary to refer to coroutines that are
based on generator syntax. *coroutine* is used in contexts where both
definitions are applicable.
New Coroutine Declaration Syntax
--------------------------------
The following new syntax is used to declare a *native coroutine*::
async def read_data(db):
pass
Key properties of *native coroutines*:
* ``async def`` functions are always coroutines, even if they do not
contain ``await`` expressions.
* It is a ``SyntaxError`` to have ``yield`` or ``yield from``
expressions in an ``async`` function.
* Internally, two new code object flags were introduced:
- ``CO_COROUTINE`` is used to enable runtime detection of
*coroutines* (and migrating existing code).
- ``CO_NATIVE_COROUTINE`` is used to mark *native coroutines*
(defined with new syntax.)
All coroutines have ``CO_COROUTINE``, ``CO_NATIVE_COROUTINE``, and
``CO_GENERATOR`` flags set.
* Regular generators, when called, return a *generator object*;
similarly, coroutines return a *coroutine object*.
* ``StopIteration`` exceptions are not propagated out of coroutines,
and are replaced with a ``RuntimeError``. For regular generators
such behavior requires a future import (see PEP 479).
* See also `Coroutine objects`_ section.
types.coroutine()
-----------------
A new function ``coroutine(gen)`` is added to the ``types`` module. It
allows interoperability between existing *generator-based coroutines*
in asyncio and *native coroutines* introduced by this PEP.
The function applies ``CO_COROUTINE`` flag to generator-function's code
object, making it return a *coroutine object*.
The function can be used as a decorator, since it modifies generator-
functions in-place and returns them.
Note, that the ``CO_NATIVE_COROUTINE`` flag is not applied by
``types.coroutine()`` to make it possible to separate *native
coroutines* defined with new syntax, from *generator-based coroutines*.
Await Expression
----------------
The following new ``await`` expression is used to obtain a result of
coroutine execution::
async def read_data(db):
data = await db.fetch('SELECT ...')
...
``await``, similarly to ``yield from``, suspends execution of
``read_data`` coroutine until ``db.fetch`` *awaitable* completes and
returns the result data.
It uses the ``yield from`` implementation with an extra step of
validating its argument. ``await`` only accepts an *awaitable*, which
can be one of:
* A *native coroutine object* returned from a *native coroutine*.
* A *generator-based coroutine object* returned from a generator
decorated with ``types.coroutine()``.
* An object with an ``__await__`` method returning an iterator.
Any ``yield from`` chain of calls ends with a ``yield``. This is a
fundamental mechanism of how *Futures* are implemented. Since,
internally, coroutines are a special kind of generators, every
``await`` is suspended by a ``yield`` somewhere down the chain of
``await`` calls (please refer to PEP 3156 for a detailed
explanation.)
To enable this behavior for coroutines, a new magic method called
``__await__`` is added. In asyncio, for instance, to enable *Future*
objects in ``await`` statements, the only change is to add
``__await__ = __iter__`` line to ``asyncio.Future`` class.
Objects with ``__await__`` method are called *Future-like* objects in
the rest of this PEP.
Also, please note that ``__aiter__`` method (see its definition
below) cannot be used for this purpose. It is a different protocol,
and would be like using ``__iter__`` instead of ``__call__`` for
regular callables.
It is a ``TypeError`` if ``__await__`` returns anything but an
iterator.
* Objects defined with CPython C API with a ``tp_await`` function,
returning an iterator (similar to ``__await__`` method).
It is a ``SyntaxError`` to use ``await`` outside of an ``async def``
function (like it is a ``SyntaxError`` to use ``yield`` outside of
``def`` function.)
It is a ``TypeError`` to pass anything other than an *awaitable* object
to an ``await`` expression.
Updated operator precedence table
'''''''''''''''''''''''''''''''''
``await`` keyword is defined as follows::
power ::= await ["**" u_expr]
await ::= ["await"] primary
where "primary" represents the most tightly bound operations of the
language. Its syntax is::
primary ::= atom | attributeref | subscription | slicing | call
See Python Documentation [12]_ and `Grammar Updates`_ section of this
proposal for details.
The key ``await`` difference from ``yield`` and ``yield from``
operators is that *await expressions* do not require parentheses around
them most of the times.
Also, ``yield from`` allows any expression as its argument, including
expressions like ``yield from a() + b()``, that would be parsed as
``yield from (a() + b())``, which is almost always a bug. In general,
the result of any arithmetic operation is not an *awaitable* object.
To avoid this kind of mistakes, it was decided to make ``await``
precedence lower than ``[]``, ``()``, and ``.``, but higher than ``**``
operators.
+------------------------------+-----------------------------------+
| Operator | Description |
+==============================+===================================+
| ``yield`` ``x``, | Yield expression |
| ``yield from`` ``x`` | |
+------------------------------+-----------------------------------+
| ``lambda`` | Lambda expression |
+------------------------------+-----------------------------------+
| ``if`` -- ``else`` | Conditional expression |
+------------------------------+-----------------------------------+
| ``or`` | Boolean OR |
+------------------------------+-----------------------------------+
| ``and`` | Boolean AND |
+------------------------------+-----------------------------------+
| ``not`` ``x`` | Boolean NOT |
+------------------------------+-----------------------------------+
| ``in``, ``not in``, | Comparisons, including membership |
| ``is``, ``is not``, ``<``, | tests and identity tests |
| ``<=``, ``>``, ``>=``, | |
| ``!=``, ``==`` | |
+------------------------------+-----------------------------------+
| ``|`` | Bitwise OR |
+------------------------------+-----------------------------------+
| ``^`` | Bitwise XOR |
+------------------------------+-----------------------------------+
| ``&`` | Bitwise AND |
+------------------------------+-----------------------------------+
| ``<<``, ``>>`` | Shifts |
+------------------------------+-----------------------------------+
| ``+``, ``-`` | Addition and subtraction |
+------------------------------+-----------------------------------+
| ``*``, ``@``, ``/``, ``//``, | Multiplication, matrix |
| ``%`` | multiplication, division, |
| | remainder |
+------------------------------+-----------------------------------+
| ``+x``, ``-x``, ``~x`` | Positive, negative, bitwise NOT |
+------------------------------+-----------------------------------+
| ``**`` | Exponentiation |
+------------------------------+-----------------------------------+
| ``await`` ``x`` | Await expression |
+------------------------------+-----------------------------------+
| ``x[index]``, | Subscription, slicing, |
| ``x[index:index]``, | call, attribute reference |
| ``x(arguments...)``, | |
| ``x.attribute`` | |
+------------------------------+-----------------------------------+
| ``(expressions...)``, | Binding or tuple display, |
| ``[expressions...]``, | list display, |
| ``{key: value...}``, | dictionary display, |
| ``{expressions...}`` | set display |
+------------------------------+-----------------------------------+
Examples of "await" expressions
'''''''''''''''''''''''''''''''
Valid syntax examples:
================================== ==================================
Expression Will be parsed as
================================== ==================================
``if await fut: pass`` ``if (await fut): pass``
``if await fut + 1: pass`` ``if (await fut) + 1: pass``
``pair = await fut, 'spam'`` ``pair = (await fut), 'spam'``
``with await fut, open(): pass`` ``with (await fut), open(): pass``
``await foo()['spam'].baz()()`` ``await ( foo()['spam'].baz()() )``
``return await coro()`` ``return ( await coro() )``
``res = await coro() ** 2`` ``res = (await coro()) ** 2``
``func(a1=await coro(), a2=0)`` ``func(a1=(await coro()), a2=0)``
``await foo() + await bar()`` ``(await foo()) + (await bar())``
``-await foo()`` ``-(await foo())``
================================== ==================================
Invalid syntax examples:
================================== ==================================
Expression Should be written as
================================== ==================================
``await await coro()`` ``await (await coro())``
``await -coro()`` ``await (-coro())``
================================== ==================================
Asynchronous Context Managers and "async with"
----------------------------------------------
An *asynchronous context manager* is a context manager that is able to
suspend execution in its *enter* and *exit* methods.
To make this possible, a new protocol for asynchronous context managers
is proposed. Two new magic methods are added: ``__aenter__`` and
``__aexit__``. Both must return an *awaitable*.
An example of an asynchronous context manager::
class AsyncContextManager:
async def __aenter__(self):
await log('entering context')
async def __aexit__(self, exc_type, exc, tb):
await log('exiting context')
New Syntax
''''''''''
A new statement for asynchronous context managers is proposed::
async with EXPR as VAR:
BLOCK
which is semantically equivalent to::
mgr = (EXPR)
aexit = type(mgr).__aexit__
aenter = type(mgr).__aenter__(mgr)
exc = True
try:
VAR = await aenter
BLOCK
except:
if not await aexit(mgr, *sys.exc_info()):
raise
else:
await aexit(mgr, None, None, None)
As with regular ``with`` statements, it is possible to specify multiple
context managers in a single ``async with`` statement.
It is an error to pass a regular context manager without ``__aenter__``
and ``__aexit__`` methods to ``async with``. It is a ``SyntaxError``
to use ``async with`` outside of an ``async def`` function.
Example
'''''''
With *asynchronous context managers* it is easy to implement proper
database transaction managers for coroutines::
async def commit(session, data):
...
async with session.transaction():
...
await session.update(data)
...
Code that needs locking also looks lighter::
async with lock:
...
instead of::
with (yield from lock):
...
Asynchronous Iterators and "async for"
--------------------------------------
An *asynchronous iterable* is able to call asynchronous code in its
*iter* implementation, and *asynchronous iterator* can call
asynchronous code in its *next* method. To support asynchronous
iteration:
1. An object must implement an ``__aiter__`` method returning an
*awaitable* resulting in an *asynchronous iterator object*.
2. An *asynchronous iterator object* must implement an ``__anext__``
method returning an *awaitable*.
3. To stop iteration ``__anext__`` must raise a ``StopAsyncIteration``
exception.
An example of asynchronous iterable::
class AsyncIterable:
async def __aiter__(self):
return self
async def __anext__(self):
data = await self.fetch_data()
if data:
return data
else:
raise StopAsyncIteration
async def fetch_data(self):
...
New Syntax
''''''''''
A new statement for iterating through asynchronous iterators is
proposed::
async for TARGET in ITER:
BLOCK
else:
BLOCK2
which is semantically equivalent to::
iter = (ITER)
iter = await type(iter).__aiter__(iter)
running = True
while running:
try:
TARGET = await type(iter).__anext__(iter)
except StopAsyncIteration:
running = False
else:
BLOCK
else:
BLOCK2
It is a ``TypeError`` to pass a regular iterable without ``__aiter__``
method to ``async for``. It is a ``SyntaxError`` to use ``async for``
outside of an ``async def`` function.
As for with regular ``for`` statement, ``async for`` has an optional
``else`` clause.
Example 1
'''''''''
With asynchronous iteration protocol it is possible to asynchronously
buffer data during iteration::
async for data in cursor:
...
Where ``cursor`` is an asynchronous iterator that prefetches ``N`` rows
of data from a database after every ``N`` iterations.
The following code illustrates new asynchronous iteration protocol::
class Cursor:
def __init__(self):
self.buffer = collections.deque()
def _prefetch(self):
...
async def __aiter__(self):
return self
async def __anext__(self):
if not self.buffer:
self.buffer = await self._prefetch()
if not self.buffer:
raise StopAsyncIteration
return self.buffer.popleft()
then the ``Cursor`` class can be used as follows::
async for row in Cursor():
print(row)
which would be equivalent to the following code::
i = await Cursor().__aiter__()
while True:
try:
row = await i.__anext__()
except StopAsyncIteration:
break
else:
print(row)
Example 2
'''''''''
The following is a utility class that transforms a regular iterable to
an asynchronous one. While this is not a very useful thing to do, the
code illustrates the relationship between regular and asynchronous
iterators.
::
class AsyncIteratorWrapper:
def __init__(self, obj):
self._it = iter(obj)
async def __aiter__(self):
return self
async def __anext__(self):
try:
value = next(self._it)
except StopIteration:
raise StopAsyncIteration
return value
async for letter in AsyncIteratorWrapper("abc"):
print(letter)
Why StopAsyncIteration?
'''''''''''''''''''''''
Coroutines are still based on generators internally. So, before PEP
479, there was no fundamental difference between
::
def g1():
yield from fut
return 'spam'
and
::
def g2():
yield from fut
raise StopIteration('spam')
And since PEP 479 is accepted and enabled by default for coroutines,
the following example will have its ``StopIteration`` wrapped into a
``RuntimeError``
::
async def a1():
await fut
raise StopIteration('spam')
The only way to tell the outside code that the iteration has ended is
to raise something other than ``StopIteration``. Therefore, a new
built-in exception class ``StopAsyncIteration`` was added.
Moreover, with semantics from PEP 479, all ``StopIteration`` exceptions
raised in coroutines are wrapped in ``RuntimeError``.
Coroutine objects
-----------------
Differences from generators
'''''''''''''''''''''''''''
This section applies only to *native coroutines* with
``CO_NATIVE_COROUTINE`` flag, i.e. defined with the new ``async def``
syntax.
**The behavior of existing *generator-based coroutines* in asyncio
remains unchanged.**
Great effort has been made to make sure that coroutines and
generators are treated as distinct concepts:
1. *Native coroutine objects* do not implement ``__iter__`` and
``__next__`` methods. Therefore, they cannot be iterated over or
passed to ``iter()``, ``list()``, ``tuple()`` and other built-ins.
They also cannot be used in a ``for..in`` loop.
An attempt to use ``__iter__`` or ``__next__`` on a *native
coroutine object* will result in a ``TypeError``.
2. *Plain generators* cannot ``yield from`` *native coroutine objects*:
doing so will result in a ``TypeError``.
3. *generator-based coroutines* (for asyncio code must be decorated
with ``(a)asyncio.coroutine``) can ``yield from`` *native coroutine
objects*.
4. ``inspect.isgenerator()`` and ``inspect.isgeneratorfunction()``
return ``False`` for *native coroutine objects* and *native
coroutine functions*.
Coroutine object methods
''''''''''''''''''''''''
Coroutines are based on generators internally, thus they share the
implementation. Similarly to generator objects, coroutine objects have
``throw()``, ``send()`` and ``close()`` methods. ``StopIteration`` and
``GeneratorExit`` play the same role for coroutine objects (although
PEP 479 is enabled by default for coroutines). See PEP 342, PEP 380,
and Python Documentation [11]_ for details.
``throw()``, ``send()`` methods for coroutine objects are used to push
values and raise errors into *Future-like* objects.
Debugging Features
------------------
A common beginner mistake is forgetting to use ``yield from`` on
coroutines::
@asyncio.coroutine
def useful():
asyncio.sleep(1) # this will do noting without 'yield from'
For debugging this kind of mistakes there is a special debug mode in
asyncio, in which ``@coroutine`` decorator wraps all functions with a
special object with a destructor logging a warning. Whenever a wrapped
generator gets garbage collected, a detailed logging message is
generated with information about where exactly the decorator function
was defined, stack trace of where it was collected, etc. Wrapper
object also provides a convenient ``__repr__`` function with detailed
information about the generator.
The only problem is how to enable these debug capabilities. Since
debug facilities should be a no-op in production mode, ``@coroutine``
decorator makes the decision of whether to wrap or not to wrap based on
an OS environment variable ``PYTHONASYNCIODEBUG``. This way it is
possible to run asyncio programs with asyncio's own functions
instrumented. ``EventLoop.set_debug``, a different debug facility, has
no impact on ``@coroutine`` decorator's behavior.
With this proposal, coroutines is a native, distinct from generators,
concept. New methods ``set_coroutine_wrapper`` and
``get_coroutine_wrapper`` are added to the ``sys`` module, with which
frameworks can provide advanced debugging facilities.
It is also important to make coroutines as fast and efficient as
possible, therefore there are no debug features enabled by default.
Example::
async def debug_me():
await asyncio.sleep(1)
def async_debug_wrap(generator):
return asyncio.CoroWrapper(generator)
sys.set_coroutine_wrapper(async_debug_wrap)
debug_me() # <- this line will likely GC the coroutine object and
# trigger asyncio.CoroWrapper's code.
assert isinstance(debug_me(), asyncio.CoroWrapper)
sys.set_coroutine_wrapper(None) # <- this unsets any
# previously set wrapper
assert not isinstance(debug_me(), asyncio.CoroWrapper)
New Standard Library Functions
------------------------------
* ``types.coroutine(gen)``. See `types.coroutine()`_ section for
details.
* ``inspect.iscoroutine(obj)`` returns ``True`` if ``obj`` is a
*coroutine object*.
* ``inspect.iscoroutinefunction(obj)`` returns ``True`` if ``obj`` is a
*coroutine function*.
* ``inspect.isawaitable(obj)`` returns ``True`` if ``obj`` can be used
in ``await`` expression. See `Await Expression`_ for details.
* ``sys.set_coroutine_wrapper(wraper)`` allows to intercept creation of
*coroutine objects*. ``wraper`` must be a callable that accepts one
argument: a *coroutine object* or ``None``. ``None`` resets the
wrapper. If called twice, the new wrapper replaces the previous one.
See `Debugging Features`_ for more details.
* ``sys.get_coroutine_wrapper()`` returns the current wrapper object.
Returns ``None`` if no wrapper was set. See `Debugging Features`_
for more details.
Glossary
========
:Native coroutine:
A coroutine function is declared with ``async def``. It uses
``await`` and ``return value``; see `New Coroutine Declaration
Syntax`_ for details.
:Native coroutine object:
Returned from a native coroutine function. See `Await Expression`_
for details.
:Generator-based coroutine:
Coroutines based on generator syntax. Most common example are
functions decorated with ``(a)asyncio.coroutine``.
:Generator-based coroutine object:
Returned from a generator-based coroutine function.
:Coroutine:
Either *native coroutine* or *generator-based coroutine*.
:Coroutine object:
Either *native coroutine object* or *generator-based coroutine
object*.
:Future-like object:
An object with an ``__await__`` method, or a C object with
``tp_await`` function, returning an iterator. Can be consumed by
an ``await`` expression in a coroutine. A coroutine waiting for a
Future-like object is suspended until the Future-like object's
``__await__`` completes, and returns the result. See `Await
Expression`_ for details.
:Awaitable:
A *Future-like* object or a *coroutine object*. See `Await
Expression`_ for details.
:Asynchronous context manager:
An asynchronous context manager has ``__aenter__`` and ``__aexit__``
methods and can be used with ``async with``. See `Asynchronous
Context Managers and "async with"`_ for details.
:Asynchronous iterable:
An object with an ``__aiter__`` method, which must return an
*asynchronous iterator* object. Can be used with ``async for``.
See `Asynchronous Iterators and "async for"`_ for details.
:Asynchronous iterator:
An asynchronous iterator has an ``__anext__`` method. See
`Asynchronous Iterators and "async for"`_ for details.
List of functions and methods
=============================
================= =================================== =================
Method Can contain Can't contain
================= =================================== =================
async def func await, return value yield, yield from
async def __a*__ await, return value yield, yield from
def __a*__ return awaitable await
def __await__ yield, yield from, return iterable await
generator yield, yield from, return value await
================= =================================== =================
Where:
* "async def func": native coroutine;
* "async def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``,
``__aexit__`` defined with the ``async`` keyword;
* "def __a*__": ``__aiter__``, ``__anext__``, ``__aenter__``,
``__aexit__`` defined without the ``async`` keyword, must return an
*awaitable*;
* "def __await__": ``__await__`` method to implement *Future-like*
objects;
* generator: a "regular" generator, function defined with ``def`` and
which contains a least one ``yield`` or ``yield from`` expression.
Transition Plan
===============
To avoid backwards compatibility issues with ``async`` and ``await``
keywords, it was decided to modify ``tokenizer.c`` in such a way, that
it:
* recognizes ``async def`` name tokens combination (start of a
native coroutine);
* keeps track of regular functions and native coroutines;
* replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with
``AWAIT`` when in the process of yielding tokens for native
coroutines.
This approach allows for seamless combination of new syntax features
(all of them available only in ``async`` functions) with any existing
code.
An example of having "async def" and "async" attribute in one piece of
code::
class Spam:
async = 42
async def ham():
print(getattr(Spam, 'async'))
# The coroutine can be executed and will print '42'
Backwards Compatibility
-----------------------
This proposal preserves 100% backwards compatibility.
asyncio
-------
``asyncio`` module was adapted and tested to work with coroutines and
new statements. Backwards compatibility is 100% preserved, i.e. all
existing code will work as-is.
The required changes are mainly:
1. Modify ``(a)asyncio.coroutine`` decorator to use new
``types.coroutine()`` function.
2. Add ``__await__ = __iter__`` line to ``asyncio.Future`` class.
3. Add ``ensure_task()`` as an alias for ``async()`` function.
Deprecate ``async()`` function.
Migration strategy
''''''''''''''''''
Because *plain generators* cannot ``yield from`` *native coroutine
objects* (see `Differences from generators`_ section for more details),
it is advised to make sure that all generator-based coroutines are
decorated with ``(a)asyncio.coroutine`` *before* starting to use the new
syntax.
Grammar Updates
---------------
Grammar changes are also fairly minimal::
decorated: decorators (classdef | funcdef | async_funcdef)
async_funcdef: ASYNC funcdef
compound_stmt: (if_stmt | while_stmt | for_stmt | try_stmt | with_stmt
| funcdef | classdef | decorated | async_stmt)
async_stmt: ASYNC (funcdef | with_stmt | for_stmt)
power: atom_expr ['**' factor]
atom_expr: [AWAIT] atom trailer*
Transition Period Shortcomings
------------------------------
There is just one.
Until ``async`` and ``await`` are not proper keywords, it is not
possible (or at least very hard) to fix ``tokenizer.c`` to recognize
them on the **same line** with ``def`` keyword::
# async and await will always be parsed as variables
async def outer(): # 1
def nested(a=(await fut)):
pass
async def foo(): return (await fut) # 2
Since ``await`` and ``async`` in such cases are parsed as ``NAME``
tokens, a ``SyntaxError`` will be raised.
To workaround these issues, the above examples can be easily rewritten
to a more readable form::
async def outer(): # 1
a_default = await fut
def nested(a=a_default):
pass
async def foo(): # 2
return (await fut)
This limitation will go away as soon as ``async`` and ``await`` are
proper keywords. Or if it's decided to use a future import for this
PEP.
Deprecation Plans
-----------------
``async`` and ``await`` names will be softly deprecated in CPython 3.5
and 3.6. In 3.7 we will transform them to proper keywords. Making
``async`` and ``await`` proper keywords before 3.7 might make it harder
for people to port their code to Python 3.
Design Considerations
=====================
PEP 3152
--------
PEP 3152 by Gregory Ewing proposes a different mechanism for coroutines
(called "cofunctions"). Some key points:
1. A new keyword ``codef`` to declare a *cofunction*. *Cofunction* is
always a generator, even if there is no ``cocall`` expressions
inside it. Maps to ``async def`` in this proposal.
2. A new keyword ``cocall`` to call a *cofunction*. Can only be used
inside a *cofunction*. Maps to ``await`` in this proposal (with
some differences, see below.)
3. It is not possible to call a *cofunction* without a ``cocall``
keyword.
4. ``cocall`` grammatically requires parentheses after it::
atom: cocall | <existing alternatives for atom>
cocall: 'cocall' atom cotrailer* '(' [arglist] ')'
cotrailer: '[' subscriptlist ']' | '.' NAME
5. ``cocall f(*args, **kwds)`` is semantically equivalent to
``yield from f.__cocall__(*args, **kwds)``.
Differences from this proposal:
1. There is no equivalent of ``__cocall__`` in this PEP, which is
called and its result is passed to ``yield from`` in the ``cocall``
expression. ``await`` keyword expects an *awaitable* object,
validates the type, and executes ``yield from`` on it. Although,
``__await__`` method is similar to ``__cocall__``, but is only used
to define *Future-like* objects.
2. ``await`` is defined in almost the same way as ``yield from`` in the
grammar (it is later enforced that ``await`` can only be inside
``async def``). It is possible to simply write ``await future``,
whereas ``cocall`` always requires parentheses.
3. To make asyncio work with PEP 3152 it would be required to modify
``(a)asyncio.coroutine`` decorator to wrap all functions in an object
with a ``__cocall__`` method, or to implement ``__cocall__`` on
generators. To call *cofunctions* from existing generator-based
coroutines it would be required to use ``costart(cofunc, *args,
**kwargs)`` built-in.
4. Since it is impossible to call a *cofunction* without a ``cocall``
keyword, it automatically prevents the common mistake of forgetting
to use ``yield from`` on generator-based coroutines. This proposal
addresses this problem with a different approach, see `Debugging
Features`_.
5. A shortcoming of requiring a ``cocall`` keyword to call a coroutine
is that if is decided to implement coroutine-generators --
coroutines with ``yield`` or ``async yield`` expressions -- we
wouldn't need a ``cocall`` keyword to call them. So we'll end up
having ``__cocall__`` and no ``__call__`` for regular coroutines,
and having ``__call__`` and no ``__cocall__`` for coroutine-
generators.
6. Requiring parentheses grammatically also introduces a whole lot
of new problems.
The following code::
await fut
await function_returning_future()
await asyncio.gather(coro1(arg1, arg2), coro2(arg1, arg2))
would look like::
cocall fut() # or cocall costart(fut)
cocall (function_returning_future())()
cocall asyncio.gather(costart(coro1, arg1, arg2),
costart(coro2, arg1, arg2))
7. There are no equivalents of ``async for`` and ``async with`` in PEP
3152.
Coroutine-generators
--------------------
With ``async for`` keyword it is desirable to have a concept of a
*coroutine-generator* -- a coroutine with ``yield`` and ``yield from``
expressions. To avoid any ambiguity with regular generators, we would
likely require to have an ``async`` keyword before ``yield``, and
``async yield from`` would raise a ``StopAsyncIteration`` exception.
While it is possible to implement coroutine-generators, we believe that
they are out of scope of this proposal. It is an advanced concept that
should be carefully considered and balanced, with a non-trivial changes
in the implementation of current generator objects. This is a matter
for a separate PEP.
Why "async" and "await" keywords
--------------------------------
async/await is not a new concept in programming languages:
* C# has it since long time ago [5]_;
* proposal to add async/await in ECMAScript 7 [2]_;
see also Traceur project [9]_;
* Facebook's Hack/HHVM [6]_;
* Google's Dart language [7]_;
* Scala [8]_;
* proposal to add async/await to C++ [10]_;
* and many other less popular languages.
This is a huge benefit, as some users already have experience with
async/await, and because it makes working with many languages in one
project easier (Python with ECMAScript 7 for instance).
Why "__aiter__" returns awaitable
---------------------------------
In principle, ``__aiter__`` could be a regular function. There are
several good reasons to make it a coroutine:
* as most of the ``__anext__``, ``__aenter__``, and ``__aexit__``
methods are coroutines, users would often make a mistake defining it
as ``async`` anyways;
* there might be a need to run some asynchronous operations in
``__aiter__``, for instance to prepare DB queries or do some file
operation.
Importance of "async" keyword
-----------------------------
While it is possible to just implement ``await`` expression and treat
all functions with at least one ``await`` as coroutines, this approach
makes APIs design, code refactoring and its long time support harder.
Let's pretend that Python only has ``await`` keyword::
def useful():
...
await log(...)
...
def important():
await useful()
If ``useful()`` function is refactored and someone removes all
``await`` expressions from it, it would become a regular python
function, and all code that depends on it, including ``important()``
would be broken. To mitigate this issue a decorator similar to
``(a)asyncio.coroutine`` has to be introduced.
Why "async def"
---------------
For some people bare ``async name(): pass`` syntax might look more
appealing than ``async def name(): pass``. It is certainly easier to
type. But on the other hand, it breaks the symmetry between ``async
def``, ``async with`` and ``async for``, where ``async`` is a modifier,
stating that the statement is asynchronous. It is also more consistent
with the existing grammar.
Why not "await for" and "await with"
------------------------------------
``async`` is an adjective, and hence it is a better choice for a
*statement qualifier* keyword. ``await for/with`` would imply that
something is awaiting for a completion of a ``for`` or ``with``
statement.
Why "async def" and not "def async"
-----------------------------------
``async`` keyword is a *statement qualifier*. A good analogy to it are
"static", "public", "unsafe" keywords from other languages. "async
for" is an asynchronous "for" statement, "async with" is an
asynchronous "with" statement, "async def" is an asynchronous function.
Having "async" after the main statement keyword might introduce some
confusion, like "for async item in iterator" can be read as "for each
asynchronous item in iterator".
Having ``async`` keyword before ``def``, ``with`` and ``for`` also
makes the language grammar simpler. And "async def" better separates
coroutines from regular functions visually.
Why not a __future__ import
---------------------------
``__future__`` imports are inconvenient and easy to forget to add.
Also, they are enabled for the whole source file. Consider that there
is a big project with a popular module named "async.py". With future
imports it is required to either import it using ``__import__()`` or
``importlib.import_module()`` calls, or to rename the module. The
proposed approach makes it possible to continue using old code and
modules without a hassle, while coming up with a migration plan for
future python versions.
Why magic methods start with "a"
--------------------------------
New asynchronous magic methods ``__aiter__``, ``__anext__``,
``__aenter__``, and ``__aexit__`` all start with the same prefix "a".
An alternative proposal is to use "async" prefix, so that ``__aiter__``
becomes ``__async_iter__``. However, to align new magic methods with
the existing ones, such as ``__radd__`` and ``__iadd__`` it was decided
to use a shorter version.
Why not reuse existing magic names
----------------------------------
An alternative idea about new asynchronous iterators and context
managers was to reuse existing magic methods, by adding an ``async``
keyword to their declarations::
class CM:
async def __enter__(self): # instead of __aenter__
...
This approach has the following downsides:
* it would not be possible to create an object that works in both
``with`` and ``async with`` statements;
* it would break backwards compatibility, as nothing prohibits from
returning a Future-like objects from ``__enter__`` and/or
``__exit__`` in Python <= 3.4;
* one of the main points of this proposal is to make native coroutines
as simple and foolproof as possible, hence the clear separation of
the protocols.
Why not reuse existing "for" and "with" statements
--------------------------------------------------
The vision behind existing generator-based coroutines and this proposal
is to make it easy for users to see where the code might be suspended.
Making existing "for" and "with" statements to recognize asynchronous
iterators and context managers will inevitably create implicit suspend
points, making it harder to reason about the code.
Comprehensions
--------------
Syntax for asynchronous comprehensions could be provided, but this
construct is outside of the scope of this PEP.
Async lambda functions
----------------------
Syntax for asynchronous lambda functions could be provided, but this
construct is outside of the scope of this PEP.
Performance
===========
Overall Impact
--------------
This proposal introduces no observable performance impact. Here is an
output of python's official set of benchmarks [4]_:
::
python perf.py -r -b default ../cpython/python.exe
../cpython-aw/python.exe
[skipped]
Report on Darwin ysmac 14.3.0 Darwin Kernel Version 14.3.0:
Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64
x86_64 i386
Total CPU cores: 8
### etree_iterparse ###
Min: 0.365359 -> 0.349168: 1.05x faster
Avg: 0.396924 -> 0.379735: 1.05x faster
Significant (t=9.71)
Stddev: 0.01225 -> 0.01277: 1.0423x larger
The following not significant results are hidden, use -v to show them:
django_v2, 2to3, etree_generate, etree_parse, etree_process,
fastpickle,
fastunpickle, json_dump_v2, json_load, nbody, regex_v8, tornado_http.
Tokenizer modifications
-----------------------
There is no observable slowdown of parsing python files with the
modified tokenizer: parsing of one 12Mb file
(``Lib/test/test_binop.py`` repeated 1000 times) takes the same amount
of time.
async/await
-----------
The following micro-benchmark was used to determine performance
difference between "async" functions and generators::
import sys
import time
def binary(n):
if n <= 0:
return 1
l = yield from binary(n - 1)
r = yield from binary(n - 1)
return l + 1 + r
async def abinary(n):
if n <= 0:
return 1
l = await abinary(n - 1)
r = await abinary(n - 1)
return l + 1 + r
def timeit(gen, depth, repeat):
t0 = time.time()
for _ in range(repeat):
list(gen(depth))
t1 = time.time()
print('{}({}) * {}: total {:.3f}s'.format(
gen.__name__, depth, repeat, t1-t0))
The result is that there is no observable performance difference.
Minimum timing of 3 runs
::
abinary(19) * 30: total 12.985s
binary(19) * 30: total 12.953s
Note that depth of 19 means 1,048,575 calls.
Reference Implementation
========================
The reference implementation can be found here: [3]_.
List of high-level changes and new protocols
--------------------------------------------
1. New syntax for defining coroutines: ``async def`` and new ``await``
keyword.
2. New ``__await__`` method for Future-like objects, and new
``tp_await`` slot in ``PyTypeObject``.
3. New syntax for asynchronous context managers: ``async with``. And
associated protocol with ``__aenter__`` and ``__aexit__`` methods.
4. New syntax for asynchronous iteration: ``async for``. And
associated protocol with ``__aiter__``, ``__aexit__`` and new built-
in exception ``StopAsyncIteration``.
5. New AST nodes: ``AsyncFunctionDef``, ``AsyncFor``, ``AsyncWith``,
``Await``.
6. New functions: ``sys.set_coroutine_wrapper(callback)``,
``sys.get_coroutine_wrapper()``, ``types.coroutine(gen)``,
``inspect.iscoroutinefunction()``, ``inspect.iscoroutine()``,
and ``inspect.isawaitable()``.
7. New ``CO_COROUTINE`` and ``CO_NATIVE_COROUTINE`` bit flags for code
objects.
While the list of changes and new things is not short, it is important
to understand, that most users will not use these features directly.
It is intended to be used in frameworks and libraries to provide users
with convenient to use and unambiguous APIs with ``async def``,
``await``, ``async for`` and ``async with`` syntax.
Working example
---------------
All concepts proposed in this PEP are implemented [3]_ and can be
tested.
::
import asyncio
async def echo_server():
print('Serving on localhost:8000')
await asyncio.start_server(handle_connection,
'localhost', 8000)
async def handle_connection(reader, writer):
print('New connection...')
while True:
data = await reader.read(8192)
if not data:
break
print('Sending {:.10}... back'.format(repr(data)))
writer.write(data)
loop = asyncio.get_event_loop()
loop.run_until_complete(echo_server())
try:
loop.run_forever()
finally:
loop.close()
References
==========
.. [1] https://docs.python.org/3/library/asyncio-task.html#asyncio.coroutine
.. [2] http://wiki.ecmascript.org/doku.php?id=strawman:async_functions
.. [3] https://github.com/1st1/cpython/tree/await
.. [4] https://hg.python.org/benchmarks
.. [5] https://msdn.microsoft.com/en-us/library/hh191443.aspx
.. [6] http://docs.hhvm.com/manual/en/hack.async.php
.. [7] https://www.dartlang.org/articles/await-async/
.. [8] http://docs.scala-lang.org/sips/pending/async.html
.. [9]
https://github.com/google/traceur-compiler/wiki/LanguageFeatures#async-func…
.. [10]
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3722.pdf (PDF)
.. [11]
https://docs.python.org/3/reference/expressions.html#generator-iterator-met…
.. [12] https://docs.python.org/3/reference/expressions.html#primaries
Acknowledgments
===============
I thank Guido van Rossum, Victor Stinner, Elvis Pranskevichus, Andrew
Svetlov, and Łukasz Langa for their initial feedback.
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
[View Less]
18
59