I think it is a relic from the distant past, when the lexer did generate NEWLINE for every blank line. I think the only case where you can still get a NEWLINE by itself is in interactive mode. This code is extremely convoluted and may be buggy in end cases; this could explain why you get a continuation prompt after entering a comment in interactive mode... --Guido On 10/6/05, Michael Hudson <mwh@python.net> wrote:

"Matthew F. Barnes" <mfb@lotusland.dyndns.org> writes:

...

I posted this question to python-help, but I think I have a better chance of getting the answer here.

I'm looking for clarification on when NEWLINE tokens are generated during lexical analysis of Python source code. In particular, I'm confused about some of the top-level components in Python's grammar (file_input, interactive_input, and eval_input).

Section 2.1.7 of the reference manual states that blank lines (lines consisting only of whitespace and possibly a comment) do not generate NEWLINE tokens. This is supported by the definition of a suite, which does not allow for standalone or consecutive NEWLINE tokens.

suite ::= stmt_list NEWLINE | NEWLINE INDENT statement+ DEDENT

I don't have the spare brain cells to think about your real problem (sorry) but something to be aware of is that the pseudo EBNF of the reference manual is purely descriptive -- it is not actually used in the parsing of Python code at all. Among other things this means it could well just be wrong :/

The real grammar is Grammar/Grammar in the source distribution.

Cheers, mwh

-- The Internet is full. Go away. -- http://www.disobey.com/devilshat/ds011101.htm _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org

-- --Guido van Rossum (home page: http://www.python.org/~guido/)