[Python-checkins] r64036 - python/trunk/Doc/library/tokenize.rst

georg.brandl python-checkins at python.org
Sun Jun 8 10:54:41 CEST 2008


Author: georg.brandl
Date: Sun Jun  8 10:54:40 2008
New Revision: 64036

Log:
#3028: tokenize passes the physical line.


Modified:
   python/trunk/Doc/library/tokenize.rst

Modified: python/trunk/Doc/library/tokenize.rst
==============================================================================
--- python/trunk/Doc/library/tokenize.rst	(original)
+++ python/trunk/Doc/library/tokenize.rst	Sun Jun  8 10:54:40 2008
@@ -15,21 +15,20 @@
 
 The primary entry point is a :term:`generator`:
 
-
 .. function:: generate_tokens(readline)
 
-   The :func:`generate_tokens` generator requires one argument, *readline*, which
-   must be a callable object which provides the same interface as the
+   The :func:`generate_tokens` generator requires one argument, *readline*,
+   which must be a callable object which provides the same interface as the
    :meth:`readline` method of built-in file objects (see section
-   :ref:`bltin-file-objects`).  Each call to the function should return one line of
-   input as a string.
+   :ref:`bltin-file-objects`).  Each call to the function should return one line
+   of input as a string.
 
    The generator produces 5-tuples with these members: the token type; the token
-   string; a 2-tuple ``(srow, scol)`` of ints specifying the row and column where
-   the token begins in the source; a 2-tuple ``(erow, ecol)`` of ints specifying
-   the row and column where the token ends in the source; and the line on which the
-   token was found. The line passed is the *logical* line; continuation lines are
-   included.
+   string; a 2-tuple ``(srow, scol)`` of ints specifying the row and column
+   where the token begins in the source; a 2-tuple ``(erow, ecol)`` of ints
+   specifying the row and column where the token ends in the source; and the
+   line on which the token was found.  The line passed is the *physical* line,
+   that is, continuation lines are not handled specially.
 
    .. versionadded:: 2.2
 


More information about the Python-checkins mailing list