[issue12943] tokenize: add python -m tokenize support back

[Meador Inge]

Meador Inge <meadori at gmail.com> added the comment:

> I don't think the help option needs to be documented, it will document 
> itself.

Normally I would document it anyway, but in this case there is only the
one option.  So, I dropped it.

> An additional suggestion is to catch errors on tokenizing and exiting 
> with a non-zero exit status in these cases.

I like that idea.  Sample output:

$ python -m tokenize foo.py
foo.py: error: unknown encoding: meador
$ python -m tokenize foo.py
foo.py:7:2: error: unindent does not match any outer indentation level
$ ./python -m tokenize bogus.py
error: [Errno 2] No such file or directory: 'bogus.py'
$ python -m tokenize foo.py
$ echo '[1,' | ./python -m tokenize
1,0-1,1:            OP             '['            
1,1-1,2:            NUMBER         '1'            
1,2-1,3:            OP             ','            
1,3-1,4:            NL             '\n'           
<stdin>:2:0: error: EOF in multi-line statement

Attached is the latest patch implementing these suggestions.

----------
Added file: http://bugs.python.org/file23228/issue12943-5.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12943>
_______________________________________