https://hg.python.org/peps/rev/552773d7e085 changeset: 5811:552773d7e085 user: Yury Selivanov <yselivanov@sprymix.com> date: Wed Apr 29 23:10:23 2015 -0400 summary: pep-0492: Reword tokenization algorithm description. files: pep-0492.txt | 16 +++++++++------- 1 files changed, 9 insertions(+), 7 deletions(-) diff --git a/pep-0492.txt b/pep-0492.txt --- a/pep-0492.txt +++ b/pep-0492.txt @@ -781,14 +781,16 @@ keywords, it was decided to modify ``tokenizer.c`` in such a way, that it: -* recognizes ``async def`` name tokens combination (start of a - native coroutine); +* recognizes ``async def`` ``NAME`` tokens combination; -* keeps track of regular functions and native coroutines; +* keeps track of regular ``def`` and ``async def`` indented blocks; -* replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with - ``AWAIT`` when in the process of yielding tokens for native - coroutines. +* while tokenizing ``async def`` block, it replaces ``'async'`` + ``NAME`` token with ``ASYNC``, and ``'await'`` ``NAME`` token with + ``AWAIT``; + +* while tokenizing ``def`` block, it yields ``'async'`` and ``'await'`` + ``NAME`` tokens as is. This approach allows for seamless combination of new syntax features (all of them available only in ``async`` functions) with any existing @@ -843,7 +845,7 @@ Grammar Updates --------------- -Grammar changes are also fairly minimal:: +Grammar changes are fairly minimal:: decorated: decorators (classdef | funcdef | async_funcdef) async_funcdef: ASYNC funcdef -- Repository URL: https://hg.python.org/peps