[Python-Dev] A minimal Python interpreter written in Python for experimenting with language changes

asrp asrp at email.com
Sat Feb 3 05:45:15 EST 2018


> Are you aware of pypy?
> 

Yes, but I do not know it well. It seems to be more focused on JIT compiling with many of its functions written for special cases and I did not find a way to make experimental changes easily.

I even tried (briefly) to have the main loop of this interpreter run in RPython but didn't find enough information on which subset of Python does RPython handle.

> Can you give an example of how you would do that? I don't mean the 
> mechanism used, I mean how would a developer implement a new syntactic 
> feature. Suppose I wanted to add a new clause to for...else, let's say:
>
>     for ... :
>        block
>     otherwise:
>        # runs only if the for-loop was empty
> 
> How would do I do that?
> 

Here's one way to do that. Sorry for the lack of doc for the moment.

To alter the language's grammar, you'd create a string with the new grammar, parse it and set the result to pyterp.parser. Then, depending on how the grammar was changed (which affects what AST is created), you'd change the corresponding function to handle the new semantics.

I didn't quite understand what you mean by "was empty". However, I noticed that "else" blocks after for are ignored in the current implement. Here's an example session adding "else".

If you tell me a bit more about the intended behaviour of "otherwise", I'd be happy to do an example with that clause.

    $ ipython -i test/python_repl.py 
    p>> simport simple_ast
    p>> for i in [1, 2]:
    ...     print i
    ... 
    1
    2
    p>> for i in [1, 2, 3]:
    ...     print i
    ... else:
    ...     print 100
    ... 
    1
    2
    3
    p>> ^D

In the above for loop, the else block is ignored! I press Control-D to exist the interpreter.

    In [1]: grammar = python_grammar.full_definition + python_grammar.extra

    In [2]: grammar += r"""
       ...: for_stmt = "for" {exprlist} "in" {testlist} ":" {suite} {(SAME_INDENT "else" ":" {suite}) | void=pass_stmt}
       ...: """

    In [3]: pyterp.parser = python.Interpreter(i3.parse("grammar", grammar))

    In [4]: pyterp.repl()

Now edit for_stmt in simple_ast.py

         except StopIteration:
+            evaluate(else_block)
             return

(If else_block was not already an (ignored) parameter, we'd have to change the `def for_stmt...` line too.) And run in the REPL we just restarted.

    p>> simple_ast.reload_module(simple_ast)
    p>> for i in [1, 2, 3]:
    ...     print i
    ... else:
    ...     print 100
    ... 
    1
    2
    3
    100
    p>> for i in [1, 2, 3]:
    ...     print i
    ...     break
    ... else:
    ...     print 100
    ... 
    1
    p>> 



Some notes:

1. I'm using the host Python interpreter to change the grammar here but that's not strictly necessary if we expose pyterp itself in the global scope.
2. I'm editing simple_ast.py and reloading because its less changes but redefining a new function and setting simple_ast.for_stmt (in other words, monkey-patching the change) should also work.

If we wanted an "otherwise" clause on top of an "else" clause, we probably want to make some provision for the runtime to distinguish between the two so I'd edit the grammar with something like

    for_stmt = "for" {exprlist} "in" {testlist} ":" {suite} {((SAME_INDENT "else" ":" {suite}) | void=pass_stmt)=else_block ((SAME_INDENT "otherwise" ":" {suite}) | void=pass_stmt)=otherwise_block}

(In this case, "otherwise" blocks have to come after the "else" block if both are present.)

asrp


More information about the Python-Dev mailing list