16 Sep
2020
16 Sep
'20
9:10 a.m.
On 16.09.2020 16:33, Benjamin Peterson wrote:
Is there any proposal that would let you add new tokens?
Not a proposal, but drafted a project a while back. By patching tokenizer and the parser at runtime (generated from lib2to3.pgen), allowed to introduce different tokens / rules, and worked like a transpiler (so that you can take the source code with creating a fake encoding, giving it as a raw source stream to the new parser with updated rules and then return the result as the decoded source); https://github.com/isidentical/freesyntax/blob/master/example2.py