[Tutor] indented grammar parsing

[Kent Johnson]

On Sun, Jan 11, 2009 at 1:11 PM, spir <denis.spir at free.fr> wrote:
> Hello,
>
> this is a rather specific question about parsing an indented grammar.
> I have a working implementation (presented below) that may be worth a critic review -- if you like
> it.
>
> First issue is: is there a way to express an indented formatfing using a common grammar
> language such as BNF (or regex)? If yes, how to do that? If not, what kind of formal grammar is
> actually such a format? Is there any standard way to express it?
>
> I guess from this text that python parsers have a pre-parse phase that generate the so-called
> INDENT and DEDENT tokens -- which are then used during actual parsing.

It's pretty common for parsers to be implemented in two phases,
lexical scanning and actual parsing. The lexical scanner recognizes
patterns in the source file and outputs a sequence of tokens. The
parser analyzes the tokens, not the actual source text. So for the
Python lexical scanner to emit INDENT and DEDENT tokens doesn't seem
that strange to me.

Kent