Testing complex new syntax

[Carl Friedrich Bolz]

Hi!

astromog at gmail.com wrote:
> I have some significantly extended syntax for Python that I need to
> create a reference implementation for. My new syntax includes new
> keywords, statements and objects that are sort of like classes but not
> really. The implementation is all possible using standard Python, but
> the implementation isn't the point of what I'm doing. Speed and having
> an extra step to run a program are not issues that I need to be
> concerned with.
> I'd like to create a preprocessor if possible, because it would
> probably be easier than implementing the changes in the interpreter. I
> could just drop in standard Python code that provides the functionality
> when I encounter a part of my extended syntax. Modifying the
> interpreter, on the other hand, sounds like it would be pretty nasty,
> even though I have experience in interpreter hacking already.
> So my question is: what's the easiest way to implement a preprocessor
> system in Python? I understand I could use the tokenize module, but
> that would still require a lot of manual parsing of the Python syntax.
> Is it possible to use any of the parser module facilities to accomplish
> this without them choking on the unknown syntax? Or, alternatively,
> would modifying the interpreter ultimately be easier?
> 

I cannot really say much about how easy it would be to just write a 
preprocessor. However, I think what you are trying to do could be done 
reasonably easy with the PyPy project:

http://codespeak.net/pypy

PyPy is an implementation of a Python interpreter written in Python. 
(Disclaimer: I am a PyPy developer). It has a quite flexible 
parser/bytecode compiler that could probably be tweaked to support your 
new syntax (especially if the new constructs can be mapped to standard 
python). Feel free to ask question on the pypy developer mailing list:

http://codespeak.net/mailman/listinfo/pypy-dev

Cheers,

Carl Friedrich Bolz