Thanks! I didn't even know about that module. Does this take into account your local changes to the tokenizer, though? I've added a new token type to Grammar/Tokens, and some code to tokenizer.c to return that token type in appropriate circumstances. I've stepped through the tokenizer in the debugger, so I think it's working. When I run -m tokenize as you suggest, I don't see my custom token type.

The devguide mentions that "Lib/tokenize.py needs changes to match changes to the tokenizer.", so I'm guessing I would have to manually repeat my changes in tokenize.py to see them, right? But what I want to see is what tokenizer.c is producing when my newly built Python binary actually reads a file.

Le 30/05/2022 à 00:59, Jack a écrit :

Hi, I'm just getting into the CPython codebase just for fun, and I've just started messing around with the tokenizer and the grammar. I was wondering, is there a way to just print out the results of the tokenizer (as in just the stream of tokens it generates) in a human readable format? It would be really helpful for debugging. Hope the question's not too basic.

python -m tokenize file.py

?

See https://docs.python.org/3/library/tokenize.html#command-line-usage

Cheers,
Jean