[Python-Dev] Lexical analysis and NEWLINE tokens
mwh at python.net
Thu Oct 6 17:07:47 CEST 2005
"Matthew F. Barnes" <mfb at lotusland.dyndns.org> writes:
> I posted this question to python-help, but I think I have a better chance
> of getting the answer here.
> I'm looking for clarification on when NEWLINE tokens are generated during
> lexical analysis of Python source code. In particular, I'm confused about
> some of the top-level components in Python's grammar (file_input,
> interactive_input, and eval_input).
> Section 2.1.7 of the reference manual states that blank lines (lines
> consisting only of whitespace and possibly a comment) do not generate
> NEWLINE tokens. This is supported by the definition of a suite, which
> does not allow for standalone or consecutive NEWLINE tokens.
> suite ::= stmt_list NEWLINE | NEWLINE INDENT statement+ DEDENT
I don't have the spare brain cells to think about your real problem
(sorry) but something to be aware of is that the pseudo EBNF of the
reference manual is purely descriptive -- it is not actually used in
the parsing of Python code at all. Among other things this means it
could well just be wrong :/
The real grammar is Grammar/Grammar in the source distribution.
The Internet is full. Go away.
More information about the Python-Dev