[issue25324] Importing tokenize modifies token

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Albert-Jan's patch should be merged prior my patch for issue30455, because it can be backported, by my patch shouldn't. But first we should compare them and find differences in results.

My patch adds COMMENT, NL and ENCODING in the documentation, Albert-Jan's patch doesn't. I don't know whether it is worth to add these names to the token module in bugfix releases, and whether it is worth to document them.

My patch wraps all names in _PyParser_TokenNames after <ERRORTOKEN> with angle parenthesis, Albert-Jan's patch inserts new names between <ERRORTOKEN> and <N_TOKENS> without angle parenthesis. I don't know what is more correct.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue25324>
_______________________________________