Thanks! I didn't even know about that module. Does this take into account your local changes to the tokenizer, though? I've added a new token type to Grammar/Tokens, and some code to tokenizer.c to return that token type in appropriate circumstances. I've stepped through the tokenizer in the debugger, so I /think /it's working. When I run -m tokenize as you suggest, I don't see my custom token type. The devguide mentions that "|Lib/tokenize.py| needs changes to match changes to the tokenizer.", so I'm guessing I would have to manually repeat my changes in tokenize.py to see them, right? But what I want to see is what tokenizer.c is producing when my newly built Python binary actually reads a file. On 30/05/2022 00:09, Jean Abou Samra wrote:
Le 30/05/2022 à 00:59, Jack a écrit :
Hi, I'm just getting into the CPython codebase just for fun, and I've just started messing around with the tokenizer and the grammar. I was wondering, is there a way to just print out the results of the tokenizer (as in just the stream of tokens it generates) in a human readable format? It would be really helpful for debugging. Hope the question's not too basic.
python -m tokenize file.py
?
See https://docs.python.org/3/library/tokenize.html#command-line-usage
Cheers, Jean