python -m tokenize in 3.x ?
Hi All, I have been investing some 'tokenize' bugs recently. As a part of that investigation I was trying to use '-m tokenize', which works great in 2.x: [meadori@motherbrain cpython]$ python2.7 -m tokenize test.py 1,0-1,5: NAME 'print' 1,6-1,21: STRING '"Hello, World!"' 1,21-1,22: NEWLINE '\n' 2,0-2,0: ENDMARKER '' In 3.x, however, the functionality has been removed and replaced with some hard-wired test code: [meadori@motherbrain cpython]$ python3 -m tokenize test.py TokenInfo(type=57 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='def', start=(1, 0), end=(1, 3), line='def parseline(self, line):') TokenInfo(type=1 (NAME), string='parseline', start=(1, 4), end=(1, 13), line='def parseline(self, line):') TokenInfo(type=53 (OP), string='(', start=(1, 13), end=(1, 14), line='def parseline(self, line):') ... Why is this? I found the commit where the functionality was removed [1], but no explanation. Any objection to adding this feature back? [1] http://hg.python.org/cpython/rev/51e24512e305/ -- # Meador
My guess that there was no specific intent -- most likely it occurred
to nobody that the main() functionality was actually useful. I'd say
it's fine to put it back, and then document it (so it won't be removed
again :-).
--Guido
On Wed, Sep 7, 2011 at 7:06 PM, Meador Inge
Hi All,
I have been investing some 'tokenize' bugs recently. As a part of that investigation I was trying to use '-m tokenize', which works great in 2.x:
[meadori@motherbrain cpython]$ python2.7 -m tokenize test.py 1,0-1,5: NAME 'print' 1,6-1,21: STRING '"Hello, World!"' 1,21-1,22: NEWLINE '\n' 2,0-2,0: ENDMARKER ''
In 3.x, however, the functionality has been removed and replaced with some hard-wired test code:
[meadori@motherbrain cpython]$ python3 -m tokenize test.py TokenInfo(type=57 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='def', start=(1, 0), end=(1, 3), line='def parseline(self, line):') TokenInfo(type=1 (NAME), string='parseline', start=(1, 4), end=(1, 13), line='def parseline(self, line):') TokenInfo(type=53 (OP), string='(', start=(1, 13), end=(1, 14), line='def parseline(self, line):') ...
Why is this? I found the commit where the functionality was removed [1], but no explanation. Any objection to adding this feature back?
[1] http://hg.python.org/cpython/rev/51e24512e305/
-- # Meador _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
participants (2)
-
Guido van Rossum
-
Meador Inge