[issue12675] tokenize module happily tokenizes code with syntax errors

[Ezio Melotti]

Changes by Ezio Melotti <ezio.melotti at gmail.com>:


----------
nosy: +ezio.melotti
versions: +Python 2.7, Python 3.2

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12675>
_______________________________________