python -m tokenize < file-to-parse.py See the comment at the top of tokenize.py. IIRC, it re-implements the tokenizer, it does not call the one used for python code. Eric On 5/29/2022 6:59 PM, Jack wrote:
Hi, I'm just getting into the CPython codebase just for fun, and I've just started messing around with the tokenizer and the grammar. I was wondering, is there a way to just print out the results of the tokenizer (as in just the stream of tokens it generates) in a human readable format? It would be really helpful for debugging. Hope the question's not too basic.
Cheers.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2ZTZBAN5... Code of Conduct: http://python.org/psf/codeofconduct/