During the discussion about assignment decorators Nick brought up the idea of adding a def-from syntax: def <NAME> from <BUILDER>(<ARGS>): ... which would effectively call <BUILDER>(<NAME>, <ARGS>, <BODY>) This is much like how meta-classes work, though this would be with functions (sort of meta-functions). Ultimately, they would amount to the same thing. The current "def" statement would just have an implicit builder. During the sprints I explored the idea of this with Nick by making a "build_from" decorator, and an exec_closure builtin to provide the full capability needed to emulate the def-from syntax with a code object. It turned out the exec_closure didn't buy much. However, one thing that became apparent in discussing this with Nick is that just passing the code object of the decorated function (or of the def body) as <BODY> doesn't buy much. To really get much bang out of this you would need to pass the AST of the body. With the AST you could manipulate it as needed before compiling. (Nick's idea) Suddenly class definitions are just a special case of def-from. If you really wanted to get crazy you could pass the raw string as <BODY> (can't blame this one on Nick). With the raw string you could put just about anything in there, like a DSL or another programming language. Then parse it however you like, and use that result to compile something else or build some data set or call some external library or whatever you care to do with raw data. It would not be restricted to parsable Python In the normal "def" context Python is hardwired to turn it into a function code object, and the implicit builder to generate the function object thereon. I am reticent to suggest adding undue complexity to the languge or adversely affect readability (potentially) if it doesn't offer plenty of increased expressive power. So, crazy raw string thing aside, how about the def-from syntax, particularly with the AST passed? Nick already indicated to me that we probably should get comfy with metaclass __prepare__ before we get any more metaprogramming, and he's probably right. I was thinking about implementing def-from as an exercise in syntax hacking, regardless. Any thoughts? Are there better syntax hacking exercises (like Raymond's "def x.y(..." or "x.y = ..." or "x.(name)"? Could def-from have a place in the future? -eric
During the discussion about assignment decorators Nick brought up the idea of adding a def-from syntax: def <NAME> from <BUILDER>(<ARGS>): ... which would effectively call <BUILDER>(<NAME>, <ARGS>, <BODY>) This is much like how meta-classes work, though this would be with functions (sort of meta-functions). Ultimately, they would amount to the same thing. The current "def" statement would just have an implicit builder. During the sprints I explored the idea of this with Nick by making a "build_from" decorator, and an exec_closure builtin to provide the full capability needed to emulate the def-from syntax with a code object. It turned out the exec_closure didn't buy much. However, one thing that became apparent in discussing this with Nick is that just passing the code object of the decorated function (or of the def body) as <BODY> doesn't buy much. To really get much bang out of this you would need to pass the AST of the body. With the AST you could manipulate it as needed before compiling. (Nick's idea) Suddenly class definitions are just a special case of def-from. If you really wanted to get crazy you could pass the raw string as <BODY> (can't blame this one on Nick). With the raw string you could put just about anything in there, like a DSL or another programming language. Then parse it however you like, and use that result to compile something else or build some data set or call some external library or whatever you care to do with raw data. It would not be restricted to parsable Python In the normal "def" context Python is hardwired to turn it into a function code object, and the implicit builder to generate the function object thereon. I am reticent to suggest adding undue complexity to the languge or adversely affect readability (potentially) if it doesn't offer plenty of increased expressive power. So, crazy raw string thing aside, how about the def-from syntax, particularly with the AST passed? Nick already indicated to me that we probably should get comfy with metaclass __prepare__ before we get any more metaprogramming, and he's probably right. I was thinking about implementing def-from as an exercise in syntax hacking, regardless. Any thoughts? Are there better syntax hacking exercises (like Raymond's "def x.y(..." or "x.y = ..." or "x.(name)"? Could def-from have a place in the future? -eric
On Tue, Mar 29, 2011 at 10:26 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
During the discussion about assignment decorators Nick brought up the idea of adding a def-from syntax:
def <NAME> from <BUILDER>(<ARGS>): ...
which would effectively call
<BUILDER>(<NAME>, <ARGS>, <BODY>)
I'm glad you went down this particular rabbit hole in so much detail. You have proved beyond a doubt that the idea is not compatible with how Python currently compiles code, since it would mean that you couldn't save the generated bytecode to a .pyc file just by parsing and compiling the source code -- either you'd have to have the runtime environment available to generate the bytecode, or you'd have to put off generating the bytecode until much later. (I knew this all along, but had a hard time explaining it to the proponents of things like this, or the "make" statement, etc. -- many people have quite a misguided idea about how dynamic Python really is, and this doesn't stop them from proposing changes that only make sense in the alternate reality they believe they live in.)
Yeah, this is a nice reduction to the absurd of the original idea; the contradiction you've arrived at proves that the original idea cannot work.
To me, the idea has always been dead. I'm glad you've provided the supporting documentation of its demise. -- --Guido van Rossum (python.org/~guido)
I appreciate your candor. You are right that the bytecode would have to live elsewhere, probably memory. Or you would have to handle the .pyc generation dynamically. namedtuples do dynamic code generation and execution, but not on the same scale as def-from would cause. Unless I misunderstand (not unlikely), it seems like your objection is that the code for the builder in a def-from would not necessarily be around yet to build the bytecode for the .pyc file. That is one of the parts of CPython that I simply haven't gotten to yet, but I have a hunch you know what you are talking about. :) Forgive me if I am wildly off, but in that case it would require a builder to be in a C module, or for it to tie into the existing mechanism CPython uses to build .pyc files and to use them. The former is what the implicit function builder would do. And that is not the only complexity it would add. This would be just one more means of meta-programming that people would have to wrap their heads around (if they wanted to use it). It would also add more complexity to the C-API. In light of all this, the benefits would have to be substantial, which is not clear to me that they are, which in reality means that it isn't worth getting into Python for now. Sometimes features have a way of coming in later when the benefits make it worth it, but I am not going to hold my breath on this one. However, the seeming flexibility of the idea is alluring. I suppose that's why it keeps coming up. -eric On Tue, Mar 29, 2011 at 11:40 AM, Guido van Rossum <guido@python.org> wrote:
Another thing I had not considered is the effect this would have on the other implementations. I have no clue as to how difficult all this would be to accomplish in the DLR, or the JVM, or pypy, or others. I have no doubt that they could find a way, but for all I know it would put an undue burden on them. Is that a criteria for Python feature consideration, the impact on other implementations of adding the feature? -eric On Tue, Mar 29, 2011 at 12:58 PM, Eric Snow <ericsnowcurrently@gmail.com>wrote:
On Tue, Mar 29, 2011 at 11:58 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
It's worse than that. The compiler that generates the .pyc file cannot have knowledge of the environment in which the code will be executed -- and that includes things like importing modules. On Tue, Mar 29, 2011 at 12:23 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
For a feature that changes the language syntax, most certainly. For a feature that adds something to the stdlib, yes, unless you are offering functionality that would simply make no sense in another implementation. (E.g. 'dis', which you mentioned before, gets a pass because it refers to the bytecode which is a CPython-exclusive feature. But it is pretty much only used for interactive debugging.) Note that things weren't always like this. But now they are. We are striving to increase compatibility between the different Python implementations so as to reduce the pain of users switching implementations. (E.g. it would be a shame if your code would run twice as fast on PyPy but you can't port it because you happen to use one little CPython-only feature.) -- --Guido van Rossum (python.org/~guido)
During the discussion about assignment decorators Nick brought up the idea of adding a def-from syntax: def <NAME> from <BUILDER>(<ARGS>): ... which would effectively call <BUILDER>(<NAME>, <ARGS>, <BODY>) This is much like how meta-classes work, though this would be with functions (sort of meta-functions). Ultimately, they would amount to the same thing. The current "def" statement would just have an implicit builder. During the sprints I explored the idea of this with Nick by making a "build_from" decorator, and an exec_closure builtin to provide the full capability needed to emulate the def-from syntax with a code object. It turned out the exec_closure didn't buy much. However, one thing that became apparent in discussing this with Nick is that just passing the code object of the decorated function (or of the def body) as <BODY> doesn't buy much. To really get much bang out of this you would need to pass the AST of the body. With the AST you could manipulate it as needed before compiling. (Nick's idea) Suddenly class definitions are just a special case of def-from. If you really wanted to get crazy you could pass the raw string as <BODY> (can't blame this one on Nick). With the raw string you could put just about anything in there, like a DSL or another programming language. Then parse it however you like, and use that result to compile something else or build some data set or call some external library or whatever you care to do with raw data. It would not be restricted to parsable Python In the normal "def" context Python is hardwired to turn it into a function code object, and the implicit builder to generate the function object thereon. I am reticent to suggest adding undue complexity to the languge or adversely affect readability (potentially) if it doesn't offer plenty of increased expressive power. So, crazy raw string thing aside, how about the def-from syntax, particularly with the AST passed? Nick already indicated to me that we probably should get comfy with metaclass __prepare__ before we get any more metaprogramming, and he's probably right. I was thinking about implementing def-from as an exercise in syntax hacking, regardless. Any thoughts? Are there better syntax hacking exercises (like Raymond's "def x.y(..." or "x.y = ..." or "x.(name)"? Could def-from have a place in the future? -eric
On Tue, Mar 29, 2011 at 10:26 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
During the discussion about assignment decorators Nick brought up the idea of adding a def-from syntax:
def <NAME> from <BUILDER>(<ARGS>): ...
which would effectively call
<BUILDER>(<NAME>, <ARGS>, <BODY>)
I'm glad you went down this particular rabbit hole in so much detail. You have proved beyond a doubt that the idea is not compatible with how Python currently compiles code, since it would mean that you couldn't save the generated bytecode to a .pyc file just by parsing and compiling the source code -- either you'd have to have the runtime environment available to generate the bytecode, or you'd have to put off generating the bytecode until much later. (I knew this all along, but had a hard time explaining it to the proponents of things like this, or the "make" statement, etc. -- many people have quite a misguided idea about how dynamic Python really is, and this doesn't stop them from proposing changes that only make sense in the alternate reality they believe they live in.)
Yeah, this is a nice reduction to the absurd of the original idea; the contradiction you've arrived at proves that the original idea cannot work.
To me, the idea has always been dead. I'm glad you've provided the supporting documentation of its demise. -- --Guido van Rossum (python.org/~guido)
I appreciate your candor. You are right that the bytecode would have to live elsewhere, probably memory. Or you would have to handle the .pyc generation dynamically. namedtuples do dynamic code generation and execution, but not on the same scale as def-from would cause. Unless I misunderstand (not unlikely), it seems like your objection is that the code for the builder in a def-from would not necessarily be around yet to build the bytecode for the .pyc file. That is one of the parts of CPython that I simply haven't gotten to yet, but I have a hunch you know what you are talking about. :) Forgive me if I am wildly off, but in that case it would require a builder to be in a C module, or for it to tie into the existing mechanism CPython uses to build .pyc files and to use them. The former is what the implicit function builder would do. And that is not the only complexity it would add. This would be just one more means of meta-programming that people would have to wrap their heads around (if they wanted to use it). It would also add more complexity to the C-API. In light of all this, the benefits would have to be substantial, which is not clear to me that they are, which in reality means that it isn't worth getting into Python for now. Sometimes features have a way of coming in later when the benefits make it worth it, but I am not going to hold my breath on this one. However, the seeming flexibility of the idea is alluring. I suppose that's why it keeps coming up. -eric On Tue, Mar 29, 2011 at 11:40 AM, Guido van Rossum <guido@python.org> wrote:
Another thing I had not considered is the effect this would have on the other implementations. I have no clue as to how difficult all this would be to accomplish in the DLR, or the JVM, or pypy, or others. I have no doubt that they could find a way, but for all I know it would put an undue burden on them. Is that a criteria for Python feature consideration, the impact on other implementations of adding the feature? -eric On Tue, Mar 29, 2011 at 12:58 PM, Eric Snow <ericsnowcurrently@gmail.com>wrote:
On Tue, Mar 29, 2011 at 11:58 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
It's worse than that. The compiler that generates the .pyc file cannot have knowledge of the environment in which the code will be executed -- and that includes things like importing modules. On Tue, Mar 29, 2011 at 12:23 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
For a feature that changes the language syntax, most certainly. For a feature that adds something to the stdlib, yes, unless you are offering functionality that would simply make no sense in another implementation. (E.g. 'dis', which you mentioned before, gets a pass because it refers to the bytecode which is a CPython-exclusive feature. But it is pretty much only used for interactive debugging.) Note that things weren't always like this. But now they are. We are striving to increase compatibility between the different Python implementations so as to reduce the pain of users switching implementations. (E.g. it would be a shame if your code would run twice as fast on PyPy but you can't port it because you happen to use one little CPython-only feature.) -- --Guido van Rossum (python.org/~guido)
participants (3)
-
Eric Snow -
Guido van Rossum -
MRAB