"Matthew F. Barnes" <mfb@lotusland.dyndns.org> writes:
I posted this question to python-help, but I think I have a better chance of getting the answer here.
I'm looking for clarification on when NEWLINE tokens are generated during lexical analysis of Python source code. In particular, I'm confused about some of the top-level components in Python's grammar (file_input, interactive_input, and eval_input).
Section 2.1.7 of the reference manual states that blank lines (lines consisting only of whitespace and possibly a comment) do not generate NEWLINE tokens. This is supported by the definition of a suite, which does not allow for standalone or consecutive NEWLINE tokens.
suite ::= stmt_list NEWLINE | NEWLINE INDENT statement+ DEDENT
I don't have the spare brain cells to think about your real problem (sorry) but something to be aware of is that the pseudo EBNF of the reference manual is purely descriptive -- it is not actually used in the parsing of Python code at all. Among other things this means it could well just be wrong :/ The real grammar is Grammar/Grammar in the source distribution. Cheers, mwh -- The Internet is full. Go away. -- http://www.disobey.com/devilshat/ds011101.htm