A minimal Python interpreter written in Python for experimenting with language changes
Hello, I don't know if this is the right place to post this. Please redirect as needed. I've made a small Python interpreter in Python with runtime AST node semantics and edit-and-continue. I thought it could make prototyping language changes more easily and visualize usage before writing them in C. Its here: https://github.com/asrp/python_terp So, for example, redefining the for_stmt function in the right scope changes the behaviour of future for loops at runtime. Although from discussion I've read in PEPs, actual implementation always look like a non-issue (which seems like magic to me) so maybe no-one here actually needs this. (I really needed edit-and-continue for one of my projects but of course, running it in this extra interpreter is much too slow.) asrp
Are you aware of pypy?
-----Original Message----- From: Python-Dev [mailto:python-dev-bounces+tritium- list=sdamon.com@python.org] On Behalf Of asrp asrp Sent: Friday, February 2, 2018 7:02 PM To: python-dev@python.org Subject: [Python-Dev] A minimal Python interpreter written in Python for experimenting with language changes
Hello,
I don't know if this is the right place to post this. Please redirect as needed.
I've made a small Python interpreter in Python with runtime AST node semantics and edit-and-continue. I thought it could make prototyping language changes more easily and visualize usage before writing them in C.
Its here: https://github.com/asrp/python_terp
So, for example, redefining the for_stmt function in the right scope changes the behaviour of future for loops at runtime.
Although from discussion I've read in PEPs, actual implementation always look like a non-issue (which seems like magic to me) so maybe no-one here actually needs this.
(I really needed edit-and-continue for one of my projects but of course, running it in this extra interpreter is much too slow.)
asrp _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium- list%40sdamon.com
Are you aware of pypy?
Yes, but I do not know it well. It seems to be more focused on JIT compiling with many of its functions written for special cases and I did not find a way to make experimental changes easily. I even tried (briefly) to have the main loop of this interpreter run in RPython but didn't find enough information on which subset of Python does RPython handle.
Can you give an example of how you would do that? I don't mean the mechanism used, I mean how would a developer implement a new syntactic feature. Suppose I wanted to add a new clause to for...else, let's say:
for ... : block otherwise: # runs only if the for-loop was empty
How would do I do that?
Here's one way to do that. Sorry for the lack of doc for the moment. To alter the language's grammar, you'd create a string with the new grammar, parse it and set the result to pyterp.parser. Then, depending on how the grammar was changed (which affects what AST is created), you'd change the corresponding function to handle the new semantics. I didn't quite understand what you mean by "was empty". However, I noticed that "else" blocks after for are ignored in the current implement. Here's an example session adding "else". If you tell me a bit more about the intended behaviour of "otherwise", I'd be happy to do an example with that clause. $ ipython -i test/python_repl.py p>> simport simple_ast p>> for i in [1, 2]: ... print i ... 1 2 p>> for i in [1, 2, 3]: ... print i ... else: ... print 100 ... 1 2 3 p>> ^D In the above for loop, the else block is ignored! I press Control-D to exist the interpreter. In [1]: grammar = python_grammar.full_definition + python_grammar.extra In [2]: grammar += r""" ...: for_stmt = "for" {exprlist} "in" {testlist} ":" {suite} {(SAME_INDENT "else" ":" {suite}) | void=pass_stmt} ...: """ In [3]: pyterp.parser = python.Interpreter(i3.parse("grammar", grammar)) In [4]: pyterp.repl() Now edit for_stmt in simple_ast.py except StopIteration: + evaluate(else_block) return (If else_block was not already an (ignored) parameter, we'd have to change the `def for_stmt...` line too.) And run in the REPL we just restarted. p>> simple_ast.reload_module(simple_ast) p>> for i in [1, 2, 3]: ... print i ... else: ... print 100 ... 1 2 3 100 p>> for i in [1, 2, 3]: ... print i ... break ... else: ... print 100 ... 1 p>> Some notes: 1. I'm using the host Python interpreter to change the grammar here but that's not strictly necessary if we expose pyterp itself in the global scope. 2. I'm editing simple_ast.py and reloading because its less changes but redefining a new function and setting simple_ast.for_stmt (in other words, monkey-patching the change) should also work. If we wanted an "otherwise" clause on top of an "else" clause, we probably want to make some provision for the runtime to distinguish between the two so I'd edit the grammar with something like for_stmt = "for" {exprlist} "in" {testlist} ":" {suite} {((SAME_INDENT "else" ":" {suite}) | void=pass_stmt)=else_block ((SAME_INDENT "otherwise" ":" {suite}) | void=pass_stmt)=otherwise_block} (In this case, "otherwise" blocks have to come after the "else" block if both are present.) asrp
On Sat, Feb 03, 2018 at 11:45:15AM +0100, asrp wrote:
Can you give an example of how you would do that? I don't mean the mechanism used, I mean how would a developer implement a new syntactic feature. Suppose I wanted to add a new clause to for...else, let's say:
for ... : block otherwise: # runs only if the for-loop was empty
How would do I do that? [...] If you tell me a bit more about the intended behaviour of "otherwise", I'd be happy to do an example with that clause.
Here's a faked session showing the sort of thing I am referring to. (Note that this is just an example, not a proposal for a new language feature.) for x in [1, 2, 3]: print(x) otherwise: print("nothing there") prints 1, 2, 3. for x in []: print(x) otherwise: print("nothing there") prints "nothing there". In other words, the otherwise block runs if, and only if, the loop iterable is empty and the for block does NOT run. Can you do something like that? -- Steve
On Tue, Feb 6, 2018 at 3:40 AM, Steven D'Aprano <steve@pearwood.info> wrote:
On Sat, Feb 03, 2018 at 11:45:15AM +0100, asrp wrote:
[...]
Here's a faked session showing the sort of thing I am referring to. (Note that this is just an example, not a proposal for a new language feature.)
for x in [1, 2, 3]: print(x) otherwise: print("nothing there")
prints 1, 2, 3.
I think you meant for x in ([1, 2, 3], ):
Message-ID: <20180206034013.GZ26553@ando.pearwood.info>
On Sat, Feb 03, 2018 at 11:45:15AM +0100, asrp wrote:
Can you give an example of how you would do that? I don't mean the mechanism used, I mean how would a developer implement a new syntactic feature. Suppose I wanted to add a new clause to for...else, let's say:
for ... : block otherwise: # runs only if the for-loop was empty
How would do I do that? [...] If you tell me a bit more about the intended behaviour of "otherwise", I'd be happy to do an example with that clause.
Here's a faked session showing the sort of thing I am referring to. (Note that this is just an example, not a proposal for a new language feature.)
for x in [1, 2, 3]: print(x) otherwise: print("nothing there")
prints 1, 2, 3.
for x in []: print(x) otherwise: print("nothing there")
prints "nothing there". In other words, the otherwise block runs if, and only if, the loop iterable is empty and the for block does NOT run.
Can you do something like that?
Oh, I see. Yes, definitely. This time I'll change for_stmt in lib/simple_ast.py beforehand (but runtime reload also works). def for_stmt(index_var, iterable, block, else_block, otherwise_block): iterator = iter(evaluate(iterable)) try: assignment(index_var, iterator.next()) except StopIteration: evaluate(otherwise_block) return while_true: __caller__['__continue__'] = __continue__ __caller__['__break__'] = __break__ evaluate(block) try: assignment(index_var, iterator.next()) except StopIteration: evaluate(else_block) return Then start the interpreter $ ipython -i test/python_repl.py p>> simport simple_ast p>> ^D [...] In [1]: grammar = python_grammar.full_definition + python_grammar.extra In [2]: grammar += r""" ...: for_stmt = "for" {exprlist} "in" {testlist} ":" {suite} {((SAME_INDENT "else" ":" {suite}) | void=pass_stmt) ((SAME_INDENT "otherwise" ":" {suite}) | void=pass_stmt)} ...: """ In [3]: pyterp.parser = python.Interpreter(i3.parse("grammar", grammar)) In [4]: pyterp.repl() p>> for x in [1, 2]: ... print(x) ... otherwise: ... print("Nothing there") ... 1 2 p>> for x in []: ... print(x) ... otherwise: ... print("Nothing there") ... Nothing there p>> for x in [1, 2]: ... print(x) ... else: ... print("Something there") ... otherwise: ... print("Nothing there") ... 1 2 Something there I noticed since my last post that the else and otherwise blocks don't need to be named if they are positional in for_stmt. The grammar change here reflects that. I've also posted an example with `until_stmt` and I talk a bit more about debugging changes made there. http://blog.asrpo.com/adding_new_statement (Nothing here is a proposed language change, just demos.) asrp
-- Steve
On Sat, Feb 03, 2018 at 12:50:11AM -0500, Terry Reedy wrote:
On 2/2/2018 7:01 PM, asrp asrp wrote:
I don't know if this is the right place to post this. Please redirect as needed.
This list is for development *of* cpython. Development *with* python in general belongs on python-list.
This list is for development of Python the language, not just CPython the interpreter. It seems to me that announcing a new Python interpreter, especially one designed for the purpose of allowing rapid experimentation with the language, is on topic for this list. -- Steve
On 03.02.18 08:12, Steven D'Aprano wrote:
On Sat, Feb 03, 2018 at 12:50:11AM -0500, Terry Reedy wrote:
On 2/2/2018 7:01 PM, asrp asrp wrote:
I don't know if this is the right place to post this. Please redirect as needed.
This list is for development *of* cpython. Development *with* python in general belongs on python-list.
This list is for development of Python the language, not just CPython the interpreter. It seems to me that announcing a new Python interpreter, especially one designed for the purpose of allowing rapid experimentation with the language, is on topic for this list.
Well spoken! -- Christian Tismer-Sperling :^) tismer@stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : https://github.com/PySide 14482 Potsdam : GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023
On Sat, Feb 03, 2018 at 01:01:30AM +0100, asrp asrp wrote:
I've made a small Python interpreter in Python with runtime AST node semantics and edit-and-continue. I thought it could make prototyping language changes more easily and visualize usage before writing them in C.
That sounds interesting.
Its here: https://github.com/asrp/python_terp
So, for example, redefining the for_stmt function in the right scope changes the behaviour of future for loops at runtime.
Can you give an example of how you would do that? I don't mean the mechanism used, I mean how would a developer implement a new syntactic feature. Suppose I wanted to add a new clause to for...else, let's say: for ... : block otherwise: # runs only if the for-loop was empty How would do I do that? (I've read the example on your github page, and it doesn't have much detail.) -- Steve
Hi user with no real name yet, On 03.02.18 01:01, asrp asrp wrote:
Hello,
I don't know if this is the right place to post this. Please redirect as needed.
I've made a small Python interpreter in Python with runtime AST node semantics and edit-and-continue. I thought it could make prototyping language changes more easily and visualize usage before writing them in C.
Its here: https://github.com/asrp/python_terp
So, for example, redefining the for_stmt function in the right scope changes the behaviour of future for loops at runtime.
Although from discussion I've read in PEPs, actual implementation always look like a non-issue (which seems like magic to me) so maybe no-one here actually needs this.
(I really needed edit-and-continue for one of my projects but of course, running it in this extra interpreter is much too slow.)
asrp
In the readme to python_terp you say: """ python_terp is intended to make language modification to Python easier to preview changes more quickly and is not intended for full CPython compatibility. However, a large subset of Python is already included. In particular, enough to run the first stage of its parser. """ This needs clarification. What do you mean by subset? A real subset or also things that are different and will stay different? To what extent are you planning to stay compatible, and where do you plan to deviate? The reason that I'm asking is that by compatible I mean the compatibility of PyPy. If you can reach that, and be it just by a subset, then it makes sense to speak of Python. Cheers - Chris -- Christian Tismer-Sperling :^) tismer@stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : https://github.com/PySide 14482 Potsdam : GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023
Hi, (My other reply to Alex and Steven doesn't seems to appear, at least not on pipermail so I hope these don't come in the wrong order.)
In the readme to python_terp you say:
""" python_terp is intended to make language modification to Python easier to preview changes more quickly and is not intended for full CPython compatibility. However, a large subset of Python is already included. In particular, enough to run the first stage of its parser. """
This needs clarification. What do you mean by subset? A real subset or also things that are different and will stay different? To what extent are you planning to stay compatible, and where do you plan to deviate?
The reason that I'm asking is that by compatible I mean the compatibility of PyPy. If you can reach that, and be it just by a subset, then it makes sense to speak of Python.
Good questions! There are two part to what I meant by that passage. One is a reflection of the current state of the interpreter and its capabilities. I put an example instead listing all differences because its easier to update but admittedly much less precise. The other concerns the intention for the future. My mind is not completely made but here's what I currently think it should/could be. First, fidelity would be determined by how the interpreter is used (if its used). More (potentially) drastic changes in the beginning than later on. Hopefully the deviations from Python (if any) for this reason does not grow too much from this. But since its for testing language changes (to Python), I don't think it would stray too much, if any. Second, I'd like to keep the whole thing small (of low complexity) as much as possible so in some cases, the less featureful choice will be kept. For example, in an earlier version, I had more functions parameters handling (defaults, *args, **kwargs, etc) but remove it because just setting up the initial local scope from the parameters passed and function definition took more than 100 lines. (Maybe I just went about this the wrong way...) However, except for easier or simpler implementation of the interpreter or some of the initial AST nodes, I don't intend to make any additions not in the Python language. One example of this is simple_for and single_if statements which I explain a bit in this old post: http://blog.asrpo.com/bootstrap_chicken_or_egg Although those are mainly intended to bootstrap the other statements like if_stmt and for_stmt and not used again later. (Well, there might be some difference for class method lookup (late-binding instead of early-binding) that's needed for edit-and-continue. But the intention would be to list all of those differences.) Feel free to ask again if I missed some aspect of your questions here. asrp
Sent: Saturday, February 03, 2018 at 12:29 PM From: "Christian Tismer" <tismer@stackless.com> To: "asrp asrp" <asrp@email.com>, python-dev@python.org Subject: Re: [Python-Dev] A minimal Python interpreter written in Python for experimenting with language changes
Hi user with no real name yet,
On 03.02.18 01:01, asrp asrp wrote:
Hello,
I don't know if this is the right place to post this. Please redirect as needed.
I've made a small Python interpreter in Python with runtime AST node semantics and edit-and-continue. I thought it could make prototyping language changes more easily and visualize usage before writing them in C.
Its here: https://github.com/asrp/python_terp
So, for example, redefining the for_stmt function in the right scope changes the behaviour of future for loops at runtime.
Although from discussion I've read in PEPs, actual implementation always look like a non-issue (which seems like magic to me) so maybe no-one here actually needs this.
(I really needed edit-and-continue for one of my projects but of course, running it in this extra interpreter is much too slow.)
asrp
In the readme to python_terp you say:
""" python_terp is intended to make language modification to Python easier to preview changes more quickly and is not intended for full CPython compatibility. However, a large subset of Python is already included. In particular, enough to run the first stage of its parser. """
This needs clarification. What do you mean by subset? A real subset or also things that are different and will stay different? To what extent are you planning to stay compatible, and where do you plan to deviate?
The reason that I'm asking is that by compatible I mean the compatibility of PyPy. If you can reach that, and be it just by a subset, then it makes sense to speak of Python.
Cheers - Chris
-- Christian Tismer-Sperling :^) tismer@stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : https://github.com/PySide 14482 Potsdam : GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023
Oh and I forgot to mention that since everything is mostly done at run time, there could just be different modules for (slightly) different languages. I don't know if that'd be problematic for maintenance though. asrp
Sent: Saturday, February 03, 2018 at 3:17 PM From: asrp <asrp@email.com> To: "Christian Tismer" <tismer@stackless.com> Cc: python-dev@python.org Subject: Re: [Python-Dev] A minimal Python interpreter written in Python for experimenting with language changes
Hi,
(My other reply to Alex and Steven doesn't seems to appear, at least not on pipermail so I hope these don't come in the wrong order.)
In the readme to python_terp you say:
""" python_terp is intended to make language modification to Python easier to preview changes more quickly and is not intended for full CPython compatibility. However, a large subset of Python is already included. In particular, enough to run the first stage of its parser. """
This needs clarification. What do you mean by subset? A real subset or also things that are different and will stay different? To what extent are you planning to stay compatible, and where do you plan to deviate?
The reason that I'm asking is that by compatible I mean the compatibility of PyPy. If you can reach that, and be it just by a subset, then it makes sense to speak of Python.
Good questions! There are two part to what I meant by that passage.
One is a reflection of the current state of the interpreter and its capabilities. I put an example instead listing all differences because its easier to update but admittedly much less precise.
The other concerns the intention for the future. My mind is not completely made but here's what I currently think it should/could be.
First, fidelity would be determined by how the interpreter is used (if its used). More (potentially) drastic changes in the beginning than later on. Hopefully the deviations from Python (if any) for this reason does not grow too much from this. But since its for testing language changes (to Python), I don't think it would stray too much, if any.
Second, I'd like to keep the whole thing small (of low complexity) as much as possible so in some cases, the less featureful choice will be kept. For example, in an earlier version, I had more functions parameters handling (defaults, *args, **kwargs, etc) but remove it because just setting up the initial local scope from the parameters passed and function definition took more than 100 lines. (Maybe I just went about this the wrong way...)
However, except for easier or simpler implementation of the interpreter or some of the initial AST nodes, I don't intend to make any additions not in the Python language. One example of this is simple_for and single_if statements which I explain a bit in this old post:
http://blog.asrpo.com/bootstrap_chicken_or_egg
Although those are mainly intended to bootstrap the other statements like if_stmt and for_stmt and not used again later.
(Well, there might be some difference for class method lookup (late-binding instead of early-binding) that's needed for edit-and-continue. But the intention would be to list all of those differences.)
Feel free to ask again if I missed some aspect of your questions here.
asrp
Sent: Saturday, February 03, 2018 at 12:29 PM From: "Christian Tismer" <tismer@stackless.com> To: "asrp asrp" <asrp@email.com>, python-dev@python.org Subject: Re: [Python-Dev] A minimal Python interpreter written in Python for experimenting with language changes
Hi user with no real name yet,
On 03.02.18 01:01, asrp asrp wrote:
Hello,
I don't know if this is the right place to post this. Please redirect as needed.
I've made a small Python interpreter in Python with runtime AST node semantics and edit-and-continue. I thought it could make prototyping language changes more easily and visualize usage before writing them in C.
Its here: https://github.com/asrp/python_terp
So, for example, redefining the for_stmt function in the right scope changes the behaviour of future for loops at runtime.
Although from discussion I've read in PEPs, actual implementation always look like a non-issue (which seems like magic to me) so maybe no-one here actually needs this.
(I really needed edit-and-continue for one of my projects but of course, running it in this extra interpreter is much too slow.)
asrp
In the readme to python_terp you say:
""" python_terp is intended to make language modification to Python easier to preview changes more quickly and is not intended for full CPython compatibility. However, a large subset of Python is already included. In particular, enough to run the first stage of its parser. """
This needs clarification. What do you mean by subset? A real subset or also things that are different and will stay different? To what extent are you planning to stay compatible, and where do you plan to deviate?
The reason that I'm asking is that by compatible I mean the compatibility of PyPy. If you can reach that, and be it just by a subset, then it makes sense to speak of Python.
Cheers - Chris
-- Christian Tismer-Sperling :^) tismer@stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : https://github.com/PySide 14482 Potsdam : GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023
participants (7)
-
Alex Walters -
asrp -
asrp asrp
-
Christian Tismer -
Steve Holden -
Steven D'Aprano -
Terry Reedy