[Python-checkins] bpo-36654: Add examples for using tokenize module programmatically (GH-12947)

Berker Peksag webhook-mailer at python.org
Sat Jan 25 14:36:09 EST 2020


https://github.com/python/cpython/commit/6dbd843dedc9e05c0e3f4714294837f0a83deebe
commit: 6dbd843dedc9e05c0e3f4714294837f0a83deebe
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Berker Peksag <berker.peksag at gmail.com>
date: 2020-01-25T22:36:04+03:00
summary:

bpo-36654: Add examples for using tokenize module programmatically (GH-12947)

(cherry picked from commit 4b09dc79f4d08d85f2cc945563e9c8ef1e531d7b)

Co-authored-by: Windson yang <wiwindson at outlook.com>

files:
M Doc/library/tokenize.rst

diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst
index 4c0a0ceef7dc4..4dd56f9e7c8bf 100644
--- a/Doc/library/tokenize.rst
+++ b/Doc/library/tokenize.rst
@@ -267,3 +267,22 @@ The exact token type names can be displayed using the :option:`-e` option:
     4,10-4,11:          RPAR           ')'
     4,11-4,12:          NEWLINE        '\n'
     5,0-5,0:            ENDMARKER      ''
+
+Example of tokenizing a file programmatically, reading unicode
+strings instead of bytes with :func:`generate_tokens`::
+
+    import tokenize
+
+    with tokenize.open('hello.py') as f:
+        tokens = tokenize.generate_tokens(f.readline)
+        for token in tokens:
+            print(token)
+
+Or reading bytes directly with :func:`.tokenize`::
+
+    import tokenize
+
+    with open('hello.py', 'rb') as f:
+        tokens = tokenize.tokenize(f.readline)
+        for token in tokens:
+            print(token)



More information about the Python-checkins mailing list