[Python-Dev] Reading Python source file

[Guido van Rossum]

On Thu, Nov 19, 2015 at 3:51 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 17.11.15 18:50, Guido van Rossum wrote:
>>
>> On Tue, Nov 17, 2015 at 8:20 AM, Serhiy Storchaka <storchaka at gmail.com>
>> wrote:
>>>
>>> Current implementation of import system went the same way. As a result
>>> importing the script as a module and running it with command line can
>>> have
>>> different behaviours in corner cases.
>>
>>
>> I'm confused. *Of course* these two behaviors differ, since Python
>> uses a different __name__. Not sure how this relates to the REPL.
>
> Sorry for confusing. I meant parser level. File parser has a few bugs, that
> can cause that the source will be differently interpreted with file and
> string parsers. For example attached script produces different output, "ä"
> if executed as a script, and "À" if imported as a module.

I see. Well, I trust you in this area (it's been too long since I
wrote the original code for all that :-).

> And there is a question about the null byte. Now compile(), exec(), eval()
> raises an exception if the script contains the null byte. Formerly they
> accepted it, but the null byte ended the script.

I like erroring out better.

> The behavior of file parser
> is more weird. The null byte makes parser to ignore the end of script
> including the newline byte [1]. E.g. "#\0\nprint('a')" is interpreted as
> "#print('a')". This is different from PyPy (and may be other
> implementations) that interprets the null byte just as ordinal character.

Yeah, this is just poorly written old code that didn't expect anyone
to care about null bytes. IMO here too the null byte should be an
error.

> The question is wherever we should support the null byte in Python sources.
>
> [1] http://bugs.python.org/issue20115

Good luck, and thanks for working on this!

-- 
--Guido van Rossum (python.org/~guido)