[Python-ideas] Does jargon make learning more difficult?

[Stephen J. Turnbull]

Abe Dillon writes:

 > That's interesting. So the parser can see one token past, for instance;
 > what would be the end of an expression, see "if", and know to expand the
 > AST?

Yes, if you want to think about it that way.

I think about it in terms of the parser receiving tokens one at a
time, using to revise the AST, until it can't.  Like a five-year-old
building a fort by piling up pillows.  She's done when she's out of
pillows, or the whole thing falls over due to imbalance.  She doesn't
have a grand plan, she just puts pillows wherever they fit.

At the point where no work can be done, the parser either has a token
in hand, which is a syntax error, or it's at end of input.  In the
latter case, it either has a complete syntactically correct AST, or
it's a syntax error.

In the end it works the same as your description, but I don't think of
it in terms of the parser knowing "where" it is in the program, but
rather in terms of whether it can do more work on the AST with the
token (or EOF) in hand.  Only when it runs out of input does it "look
at" what it's built so far to check if it's done.

If you think of expressions having their own subparser, the model is a
little more subtle.  I think of the subparsers as being coroutines
receiving tokens via (yield) from their parent.  This implies that
when the subparser actually returns an AST, the parent already has the
token that the subparser balked at.  In some sense, there's no
lookahead in the sense of something you save away to fully process
later, or pass to another actor to process.  You're always operating
on a current token.

Steve