[Python-Dev] Adding a token

[Greg Ewing]

Benjamin Peterson wrote:

> Why do you even have to add a new token? You can just put the literal
> '?' in the grammar.

I don't see how that can be sufficient. All the other tokens
have entries in the three places I mentioned, and there's
no way that pgen can generate those automatically just from
seeing a '?' in the grammar.

I just tried an experiment -- I changed the grammar to accept
'?' as an alternative to '+', and tried to use the parser
module to parse "1?2". It reported a SyntaxError.

-- 
Greg