[New-bugs-announce] [issue27582] Mispositioned SyntaxError caret for unknown code points

[Nick Coghlan]

New submission from Nick Coghlan:

Reporting by Rustom Mody on python-ideas, the SyntaxError caret is confusingly mispositioned when an invalid Unicode codepoint appears as part of a larger sequence of invalid codepoints and/or valid identifier characters:

>>> varname = “d“a”t”apoint
  File "<stdin>", line 1
    varname = “d“a”t”apoint
                          ^
SyntaxError: invalid character in identifier
>>> varname = “d“a”t”apoint.evidence
  File "<stdin>", line 1
    varname = “d“a”t”apoint.evidence
                          ^
SyntaxError: invalid character in identifier
>>> varname = “d“a”t”apoint[evidence]
  File "<stdin>", line 1
    varname = “d“a”t”apoint[evidence]
                          ^
SyntaxError: invalid character in identifier
>>> varname = “d“a”t”apoint(evidence)
  File "<stdin>", line 1
    varname = “d“a”t”apoint(evidence)
                          ^
SyntaxError: invalid character in identifier

If the invalid character is a non-identifiers ASCII character, the error message loses the trailing "in identifier" phrase and points to the correct place:

>>> varname = $d$a$t$apoint
  File "<stdin>", line 1
    varname = $d$a$t$apoint
              ^
SyntaxError: invalid syntax
>>> varname = d$a$t$apoint
  File "<stdin>", line 1
    varname = d$a$t$apoint
               ^
SyntaxError: invalid syntax
>>> varname = ^d$a$t$apoint
  File "<stdin>", line 1
    varname = ^d$a$t$apoint
              ^
SyntaxError: invalid syntax
>>> varname = !d$a$t$apoint
  File "<stdin>", line 1
    varname = !d$a$t$apoint
              ^
SyntaxError: invalid syntax
>>> varname = `d$a$t$apoint
  File "<stdin>", line 1
    varname = `d$a$t$apoint
              ^
SyntaxError: invalid syntax

----------
messages: 270914
nosy: ncoghlan
priority: normal
severity: normal
stage: needs patch
status: open
title: Mispositioned SyntaxError caret for unknown code points
type: behavior
versions: Python 3.5, Python 3.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue27582>
_______________________________________