30 May
2022
30 May
'22
10:13 a.m.
On Mon, May 30, 2022 at 1:40 AM Eric V. Smith
python -m tokenize < file-to-parse.py
See the comment at the top of tokenize.py. IIRC, it re-implements the tokenizer, it does not call the one used for python code.
Ah right, I would be surprised that there would be a public Python API
to get the tokenizer output, since there is no public C API for that
:-)
I just removed