[Python-checkins] cpython (2.7): Closes #20956: 2.7 tokenize does not produce named tuples. Patch by Sam Kimbrel.

terry.reedy python-checkins at python.org
Mon Apr 14 22:17:45 CEST 2014


http://hg.python.org/cpython/rev/d4f5a88b94b4
changeset:   90285:d4f5a88b94b4
branch:      2.7
parent:      90269:ea677e26dbeb
user:        Terry Jan Reedy <tjreedy at udel.edu>
date:        Mon Apr 14 16:17:09 2014 -0400
summary:
  Closes #20956: 2.7 tokenize does not produce named tuples. Patch by Sam Kimbrel.

files:
  Doc/library/tokenize.rst |  7 ++++---
  Misc/ACKS                |  1 +
  2 files changed, 5 insertions(+), 3 deletions(-)


diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst
--- a/Doc/library/tokenize.rst
+++ b/Doc/library/tokenize.rst
@@ -17,9 +17,10 @@
 
 To simplify token stream handling, all :ref:`operators` and :ref:`delimiters`
 tokens are returned using the generic :data:`token.OP` token type.  The exact
-type can be determined by checking the token ``string`` field on the
-:term:`named tuple` returned from :func:`tokenize.tokenize` for the character
-sequence that identifies a specific operator token.
+type can be determined by checking the second field (containing the actual
+token string matched) of the tuple returned from
+:func:`tokenize.generate_tokens` for the character sequence that identifies a
+specific operator token.
 
 The primary entry point is a :term:`generator`:
 
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -670,6 +670,7 @@
 Jason Killen
 Jan Kim
 Taek Joo Kim
+Sam Kimbrel
 W. Trevor King
 Paul Kippes
 Steve Kirsch

-- 
Repository URL: http://hg.python.org/cpython


More information about the Python-checkins mailing list