[Python-Dev] python -m tokenize in 3.x ?

[Guido van Rossum]

My guess that there was no specific intent -- most likely it occurred
to nobody that the main() functionality was actually useful. I'd say
it's fine to put it back, and then document it (so it won't be removed
again :-).

--Guido

On Wed, Sep 7, 2011 at 7:06 PM, Meador Inge <meadori at gmail.com> wrote:
> Hi All,
>
> I have been investing some 'tokenize' bugs recently.  As a part of
> that investigation I was trying to use '-m tokenize', which works
> great in 2.x:
>
> [meadori at motherbrain cpython]$ python2.7 -m tokenize test.py
> 1,0-1,5:        NAME    'print'
> 1,6-1,21:       STRING  '"Hello, World!"'
> 1,21-1,22:      NEWLINE '\n'
> 2,0-2,0:        ENDMARKER       ''
>
> In 3.x, however, the functionality has been removed and replaced with
> some hard-wired test code:
>
> [meadori at motherbrain cpython]$ python3 -m tokenize test.py
> TokenInfo(type=57 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='')
> TokenInfo(type=1 (NAME), string='def', start=(1, 0), end=(1, 3),
> line='def parseline(self, line):')
> TokenInfo(type=1 (NAME), string='parseline', start=(1, 4), end=(1,
> 13), line='def parseline(self, line):')
> TokenInfo(type=53 (OP), string='(', start=(1, 13), end=(1, 14),
> line='def parseline(self, line):')
> ...
>
> Why is this?  I found the commit where the functionality was removed
> [1], but no explanation.  Any objection to adding this feature back?
>
> [1] http://hg.python.org/cpython/rev/51e24512e305/
>
> --
> # Meador
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)