[Python-Dev] Where are universal newlines handled in the parser/compiler?

[Benjamin Peterson]

On Sat, Aug 16, 2008 at 8:34 PM, Brett Cannon <brett at python.org> wrote:
> If you import a module that uses '\r\n' line endings, Python does the
> right thing. But if you read in the bytes for the same file and then
> pass it to compile() you get an unhelpful syntax error pointing at a
> blank line.
>
> Normally I would say one should just open the source file as 'r'
> instead of 'rb', but with source code that does not work well as their
> can be a source encoding set. Lib/test/test_pep263.py is the perfect
> example of this; Windows newlines with a koi8-r encoding.
>
> What I would like to do is get compile() to work properly with a bytes
> stream just as if Python itself was handling the compilation through
> import and from a file directly. But before I try to dig into the
> parser to figure out where the translation of newlines occurs (or
> where the translation option is set), I thought I would ask to see if
> anyone just happened to know (I have already spent a few hours
> figuring out why Latin-1 encodings were not working with compile() so
> I don't want to go diving into the maze of function calls in the
> parser again).

Have a look at tok_nextc in Parser/tokenizer.c.

>
> -Brett
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.com
>



-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."