[ python-Bugs-1184112 ] Missing trailing newline with comment raises SyntaxError

[SourceForge.net]

Bugs item #1184112, was opened at 2005-04-15 18:55
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1184112&group_id=5470

Category: Parser/Compiler
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Eric Huss (ehuss)
Assigned to: Nobody/Anonymous (nobody)
Summary: Missing trailing newline with comment raises SyntaxError

Initial Comment:
The following illustrates a problem with the parser 
handling the lack of trailing newlines:

>>> parser.suite('def foo():\n\tpass\n\n# comment')
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<string>", line 4
    # comment
           ^
SyntaxError: invalid syntax
>>> parser.suite('def foo():\n\tpass\n\n# comment\n')
<parser.st object at 0x847f0a0>

This is similar to bug 501622, however, this only seems 
to happen when you have an indented block, followed by 
a comment line that has no trailing newline.

I traced through tokenizer.c and whittled down the issue 
into tok_get().  In the statement where it is processing 
the comment character and looking at the tabforms, in 
the first case this will end up with 'c' equal to EOF 
whereas in the second case "c" will eqaul '\n'.  When it 
equals EOF, it is unable to do the cleanup necessary to 
emit the DEDENT token (it immediately bails out with 
ENDMARKER which causes parsetok() to barf because 
the indentation level is still 1 inside tok_state).

Attached is a patch of a little hack I made that seems 
to fix the problem.  Although it seems to be a safe thing 
to do, it is definitely a hack.


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1184112&group_id=5470