[issue2182] tokenize: does not allow CR for a newline

[Jared Grubb]

Jared Grubb <python at jaredgrubb.com> added the comment:

Yes, but exec(string) also gives a syntax error for \r\n:

exec('x=1\r\nprint x') 

The only explanation I could find for ONLY permitting \n as newlines in
 exec(string) comes from PEP278: "There is no support for universal
newlines in strings passed to eval() or exec. It is envisioned that such
strings always have the standard \n line feed, if the strings come from
a file that file can be read with universal newlines." (This is why my
original example had to be exec(file) and not just a simple exec(string))

Of the 3 newline types, exec(*) allows 1 or all 3 as the case may be,
and tokenize allows exactly 2 of them. I honestly am not sure what the
"right" way is (or should be), but either way, the tokenize module is
not consistent with exec.

(By the way, if you're curious why I filed this issue and Issue#2180,
I'm working on the PyPy project to help improve its current Python
lexer/parser. In order to ensure that it is correct and robust, I was
experimenting with corner cases in Python syntax and I found these cases
where tokenize disagrees with exec.)

__________________________________
Tracker <report at bugs.python.org>
<http://bugs.python.org/issue2182>
__________________________________