lexical analysis of python
andrew at acooke.org
Wed Mar 11 03:35:23 CET 2009
robert.muller2 at gmail.com wrote:
> I understand the method, but when you say you "count one DEDENT for
> each level"
> well lets say you counted 3 of them. Do you have a way to interject 3
> DEDENT tokens into the token stream so that the parser receives them
> before it
> receives the next real token?
i don't know any details of your lexer, but perhaps you could emit a
single token that included the current value (DEDENTS, 3) for example.
then it would only require a very simple wrapper around the lexer to
convert that into three tokens.
in that way you would stay with a machine-generated lexer and parser, with
just a simple hand-written shim between them.
More information about the Python-list