[Python-Dev] Lexical analysis and NEWLINE tokens

[Guido van Rossum]

I think it is a relic from the distant past, when the lexer did
generate NEWLINE for every blank line. I think the only case where you
can still get a NEWLINE by itself is in interactive mode. This code is
extremely convoluted and may be buggy in end cases; this could explain
why you get a continuation prompt after entering a comment in
interactive mode...

--Guido

On 10/6/05, Michael Hudson <mwh at python.net> wrote:
> "Matthew F. Barnes" <mfb at lotusland.dyndns.org> writes:
>
> > I posted this question to python-help, but I think I have a better chance
> > of getting the answer here.
> >
> > I'm looking for clarification on when NEWLINE tokens are generated during
> > lexical analysis of Python source code.  In particular, I'm confused about
> > some of the top-level components in Python's grammar (file_input,
> > interactive_input, and eval_input).
> >
> > Section 2.1.7 of the reference manual states that blank lines (lines
> > consisting only of whitespace and possibly a comment) do not generate
> > NEWLINE tokens.  This is supported by the definition of a suite, which
> > does not allow for standalone or consecutive NEWLINE tokens.
> >
> >     suite ::= stmt_list NEWLINE | NEWLINE INDENT statement+ DEDENT
>
> I don't have the spare brain cells to think about your real problem
> (sorry) but something to be aware of is that the pseudo EBNF of the
> reference manual is purely descriptive -- it is not actually used in
> the parsing of Python code at all.  Among other things this means it
> could well just be wrong :/
>
> The real grammar is Grammar/Grammar in the source distribution.
>
> Cheers,
> mwh
>
> --
>   The Internet is full.  Go away.
>                       -- http://www.disobey.com/devilshat/ds011101.htm
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>


--
--Guido van Rossum (home page: http://www.python.org/~guido/)