[New-bugs-announce] [issue39219] Fix attributes of syntax errors raized in the tokenizer

[Serhiy Storchaka]

New submission from Serhiy Storchaka <storchaka+cpython at gmail.com>:

SyntaxError can be raised at different stages of compiling. In some places the source text is not available and should be read from the file using the file name. Which does not work in case of compiling a string or reading from stdin.

>>> 0z
  File "<stdin>", line 1
    0z
     ^
SyntaxError: invalid syntax
>>> 0xz
  File "<stdin>", line 1
SyntaxError: invalid hexadecimal literal

In the second example above the source line and the caret are absent in REPL.

The proposed PR fixes two errors in raising a SyntaxError in the tokenizer.

1. The text of the source line was not set if an exception was raised in the tokenizer. Since most of these exceptions (with more detailed description) were added in 3.8 I consider this a regressions.

2. The offset attribute was an offset in bytes. Now it is an offset in characters.

It only fixes errors in the tokenizer. There are similar bugs in other parts of the compiler.

This issue is based on the article https://aroberge.blogspot.com/2019/12/a-tiny-python-exception-oddity.html .

----------
components: Interpreter Core
messages: 359331
nosy: serhiy.storchaka
priority: normal
severity: normal
status: open
title: Fix attributes of syntax errors raized in the tokenizer
type: behavior
versions: Python 3.8, Python 3.9

_______________________________________
Python tracker <report at bugs.python.org>
<https://bugs.python.org/issue39219>
_______________________________________