[Python-Dev] Adding a token

[Greg Ewing]

I'm trying to add a '?' token to the parser, and weird things
are happening.

I've added a #define to token.h, an entry to _PyParser_TokenNames
in tokenizer.c and case for it in PyToken_OneChar(). But it's
behaving as though the tokenizer is not recognising my token.

I put in some printfs to find out whether PyToken_OneChar() is
recognising it. The results are confusing: while running
pgen, PyToken_OneChar() is being called and recognising the
new token correctly.

However, it doesn't seem to be getting called *at all* when
parsing Python code. I don't see how this can happen, because
pgen seems to use the same tokenizing code as Python itself.

Is there anything else I need to do? Does some file need
to be manually re-made?

-- 
Greg