[Python-Dev] Lexical analysis and NEWLINE tokens

[Michael Hudson]

"Matthew F. Barnes" <mfb at lotusland.dyndns.org> writes:

> I posted this question to python-help, but I think I have a better chance
> of getting the answer here.
>
> I'm looking for clarification on when NEWLINE tokens are generated during
> lexical analysis of Python source code.  In particular, I'm confused about
> some of the top-level components in Python's grammar (file_input,
> interactive_input, and eval_input).
>
> Section 2.1.7 of the reference manual states that blank lines (lines
> consisting only of whitespace and possibly a comment) do not generate
> NEWLINE tokens.  This is supported by the definition of a suite, which
> does not allow for standalone or consecutive NEWLINE tokens.
>
>     suite ::= stmt_list NEWLINE | NEWLINE INDENT statement+ DEDENT

I don't have the spare brain cells to think about your real problem
(sorry) but something to be aware of is that the pseudo EBNF of the
reference manual is purely descriptive -- it is not actually used in
the parsing of Python code at all.  Among other things this means it
could well just be wrong :/

The real grammar is Grammar/Grammar in the source distribution.

Cheers,
mwh

-- 
  The Internet is full.  Go away.
                      -- http://www.disobey.com/devilshat/ds011101.htm