[Python-checkins] r76914 - python/branches/py3k/Doc/library/tokenize.rst
ezio.melotti
python-checkins at python.org
Sun Dec 20 13:24:57 CET 2009
Author: ezio.melotti
Date: Sun Dec 20 13:24:57 2009
New Revision: 76914
Log:
Fixed capitalization and markup; added imports in the example
Modified:
python/branches/py3k/Doc/library/tokenize.rst
Modified: python/branches/py3k/Doc/library/tokenize.rst
==============================================================================
--- python/branches/py3k/Doc/library/tokenize.rst (original)
+++ python/branches/py3k/Doc/library/tokenize.rst Sun Dec 20 13:24:57 2009
@@ -94,16 +94,19 @@
(as a string) and a list of any lines (not decoded from bytes) it has read
in.
- It detects the encoding from the presence of a utf-8 bom or an encoding
- cookie as specified in pep-0263. If both a bom and a cookie are present,
+ It detects the encoding from the presence of a UTF-8 BOM or an encoding
+ cookie as specified in :pep:`263`. If both a BOM and a cookie are present,
but disagree, a SyntaxError will be raised.
- If no encoding is specified, then the default of 'utf-8' will be returned.
+ If no encoding is specified, then the default of ``'utf-8'`` will be returned.
Example of a script re-writer that transforms float literals into Decimal
objects::
+ from tokenize import tokenize, untokenize, NUMBER, STRING, NAME, OP
+ from io import BytesIO
+
def decistmt(s):
"""Substitute Decimals for floats in a string of statements.
More information about the Python-checkins
mailing list