[issue12675] tokenize module happily tokenizes code with syntax errors

[Gareth Rees]

Gareth Rees <gdr at garethrees.org> added the comment:

These errors are generated directly by the tokenizer. In tokenizer.c, the tokenizer generates ERRORTOKEN when it encounters something it can't tokenize. This causes parsetok() in parsetok.c to stop tokenizing and return an error.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue12675>
_______________________________________