[Python-checkins] peps: pep-0492: Reword tokenization algorithm description.

[yury.selivanov]

https://hg.python.org/peps/rev/552773d7e085
changeset:   5811:552773d7e085
user:        Yury Selivanov <yselivanov at sprymix.com>
date:        Wed Apr 29 23:10:23 2015 -0400
summary:
  pep-0492: Reword tokenization algorithm description.

files:
  pep-0492.txt |  16 +++++++++-------
  1 files changed, 9 insertions(+), 7 deletions(-)


diff --git a/pep-0492.txt b/pep-0492.txt
--- a/pep-0492.txt
+++ b/pep-0492.txt
@@ -781,14 +781,16 @@
 keywords, it was decided to modify ``tokenizer.c`` in such a way, that
 it:
 
-* recognizes ``async def`` name tokens combination (start of a
-  native coroutine);
+* recognizes ``async def`` ``NAME`` tokens combination;
 
-* keeps track of regular functions and native coroutines;
+* keeps track of regular ``def`` and ``async def`` indented blocks;
 
-* replaces ``'async'`` token with ``ASYNC`` and ``'await'`` token with
-  ``AWAIT`` when in the process of yielding tokens for native
-  coroutines.
+* while tokenizing ``async def`` block, it replaces ``'async'``
+  ``NAME`` token with ``ASYNC``, and ``'await'`` ``NAME`` token with
+  ``AWAIT``;
+
+* while tokenizing ``def`` block, it yields ``'async'`` and ``'await'``
+  ``NAME`` tokens as is.
 
 This approach allows for seamless combination of new syntax features
 (all of them available only in ``async`` functions) with any existing
@@ -843,7 +845,7 @@
 Grammar Updates
 ---------------
 
-Grammar changes are also fairly minimal::
+Grammar changes are fairly minimal::
 
     decorated: decorators (classdef | funcdef | async_funcdef)
     async_funcdef: ASYNC funcdef

-- 
Repository URL: https://hg.python.org/peps