[Python-Dev] Lexical analysis and NEWLINE tokens
Guido van Rossum
guido at python.org
Thu Oct 6 17:30:45 CEST 2005
I think it is a relic from the distant past, when the lexer did
generate NEWLINE for every blank line. I think the only case where you
can still get a NEWLINE by itself is in interactive mode. This code is
extremely convoluted and may be buggy in end cases; this could explain
why you get a continuation prompt after entering a comment in
On 10/6/05, Michael Hudson <mwh at python.net> wrote:
> "Matthew F. Barnes" <mfb at lotusland.dyndns.org> writes:
> > I posted this question to python-help, but I think I have a better chance
> > of getting the answer here.
> > I'm looking for clarification on when NEWLINE tokens are generated during
> > lexical analysis of Python source code. In particular, I'm confused about
> > some of the top-level components in Python's grammar (file_input,
> > interactive_input, and eval_input).
> > Section 2.1.7 of the reference manual states that blank lines (lines
> > consisting only of whitespace and possibly a comment) do not generate
> > NEWLINE tokens. This is supported by the definition of a suite, which
> > does not allow for standalone or consecutive NEWLINE tokens.
> > suite ::= stmt_list NEWLINE | NEWLINE INDENT statement+ DEDENT
> I don't have the spare brain cells to think about your real problem
> (sorry) but something to be aware of is that the pseudo EBNF of the
> reference manual is purely descriptive -- it is not actually used in
> the parsing of Python code at all. Among other things this means it
> could well just be wrong :/
> The real grammar is Grammar/Grammar in the source distribution.
> The Internet is full. Go away.
> -- http://www.disobey.com/devilshat/ds011101.htm
> Python-Dev mailing list
> Python-Dev at python.org
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
--Guido van Rossum (home page: http://www.python.org/~guido/)
More information about the Python-Dev