New Python syntax for continuing definitions for existing classes
Dear Python Developers, we have a potential idea for enhancing Python. Below you will find what the PEP might look like. A reference implementation has been written and will be posted in a follow up message. We are look forward to hearing your feedback and ideas. Kind regards, Pim Schellart & Nate Lust Abstract ======== This PEP proposes the introduction of new syntax to create a community standard, readable way to continue a definition for classes which are already defined. Rationale ========= Extending existing classes with new members currently requires the members to be defined outside of class scope and then assigned to the class, as in:: def foo(self): pass A.foo = foo del foo This pattern frequently occurs when extending classes from extension modules written in other languages, but is also not uncommon in pure Python code. This syntax is however cumbersome and error prone for the following reasons: 1. The name of the new member might clash with an existing name at module scope. 2. The developer might forget the assignment. 3. The developer might forget to delete the temporary, leaving it at module scope where it is meaningless (or worse non functional as a stand alone function). Alternatives are to use inheritance, lambda or a decorator. Inheritance is not a good option for use with Python extension modules (written in for instance C / C++). The reason is that if in the other language the inheritance relation is A<-B, and this is exposed to Python, we can't add functionality to A in Python by subclassing A<-A' without also introducing B' that inherits from B and A'. Thus one change propagates all the way down the inheritance chain. Inheritance with pure Python classes exhibit the same issue. It might be tempting to avoid (1) and (2) by subclassing, reusing the name of the base class. However this approach suffers from two issues. The first issue, like above, has to do with the inheritance relation. If A subclasses A and B subclasses A (A<-A<-B), then calls to super in B may have unexpected behavior (anything using the method resolution order may exhibit the same unexpectedness). The second issue arises when class B inherits from class A before it is extended (A<-B) and then class A is extended though inheritance (A<-A). Class A will now have new methods and data members which are not present in B, as it is a subclass only of the original A. This would be confusing to anyone examining B as it would be apparently missing members of it's parent. Adding attributes to a class using lambda is not equivalent because it only allows for expressions and not statements. A class decorator (say "@continue_class(A)" that takes methods and attributes from the class and attaches them to the wrapped type) works, but is cumbersome and requires a new class name to be introduced for each extended type. Thus suffering from the same problems numbered (1) and (2) above. A function decorator could extend a class, but also requires defining a new name and suffers from problems (1) and (2). This proposal adds the keyword combination "continue class" which instructs the interpreter to add members to an existing class. This syntax requires no new keywords and allows all existing Python code to continue to function with no modification. By combining existing keywords it avoids name clashes. Precedents for this exists in "is not" and "yield from". Another big advantage of the "continue class" syntax is discoverability. For humans, the meaning of the syntax is evident, the class will be continued with the following block. Automated tools will also benefit from this syntax as parsing a file to discover the complete definition of a class will be easier than trying to detect 'monkey patching'. Semantics ========= The following two snippets are semantically identical:: continue class A: x = 5 def foo(self): pass def bar(self): pass def foo(self): pass def bar(self): pass A.x = 5 A.foo = foo A.bar = bar del foo del bar Alternatives ============ An alternative could be to allow a function definition to include a class specifier as in:: def A.foo(self): pass Implementation ============== Adapting Python's grammar to support the continue class keyword combination requires modifying the grammar file to parse for the new keyword combination. Additional changes to Python.asdl, ast.c, compile.c, symtable.c will be required to process the parsed syntax to byte code. A reference implementation (already written and posted separately) parses the continue class line for the name of the class to be modified and loads it. Next the body of the block is compiled into a code object scoped by the name of the class. The class and the code are used as arguments to a new built-in module, called __continue_class__. This function evaluates the code block, passing the results to setattr on the supplied class.
Hello developers, I have written a functional reference implementation for the proposed continue class syntax which can be found at https://raw.githubusercontent.com/natelust/continueClassCpythonPatch/master/... . This is a mercurial patch file created against python 3.5. An example of its functionality, and demonstration that it does not interfere with the existing continue keyword is as follows: # Define a class Foo, taking a message as an argument class Foo: def __init__(self, message): self.message = message # Continue the definition of the class, adding a method to print # the class message variable continue class Foo: def second(self): print(self.message) # Create an instance of the class inst = Foo('Hello World') # Demo the continued class inst.second() # Show that existing syntax still works for i in range(5): if i == 0: continue print(i) Thank you for your time and consideration, we will be happy to take any feedback or questions on the implementation. Nate Lust On Tue, Sep 13, 2016 at 12:17 PM Pim Schellart <p.schellart@princeton.edu> wrote:
On Wed, Sep 14, 2016 at 2:22 AM, nate lust <natelust@linux.com> wrote:
AIUI, the central core to this code is: + /* Iterate over the dictionary which was populated in the eval call. + Set an attribute in the class corresponding to the key, item pair + generated by the eval statement */ + dict_size = PyDict_Size(newdict); + dictkeys = PyDict_Keys(newdict); + for (i = 0; i < dict_size; i++ ){ + key = PyList_GetItem(dictkeys, i); + item = PyDict_GetItem(newdict, key); + PyObject_SetAttr(aclass, key, item); + } How does this handle special elements such as __module__, __dict__, and __doc__? Does it overwrite them? ChrisA
On Wed, Sep 14, 2016 at 2:16 AM, Pim Schellart <p.schellart@princeton.edu> wrote:
Did you know that you can actually abuse decorators to do this with existing syntax? Check out this collection of evil uses of decorators: https://github.com/Rosuav/Decorators/blob/master/evil.py I call them "evil" because they're potentially VERY confusing, but they're not necessarily bad. The monkeypatch decorator does basically what you're doing here, but with this syntax: @monkeypatch class A: x = 5 def foo(self): pass def bar(self): pass It's a little bit magical, in that it looks up the original class using globals(); this is partly deliberate, as it means you can't accidentally monkey-patch something from the built-ins, which will either fail, or (far worse) succeed and confuse everyone. ChrisA
On 13 September 2016 at 17:29, Chris Angelico <rosuav@gmail.com> wrote:
Also, it's worth noting that while Ruby has a tradition of "open" classes (as in, the ability to add methods to class definitions after those classes have been created) Python traditionally does not encourage such use. You claim in your proposal """ This pattern frequently occurs when extending classes from extension modules written in other languages, but is also not uncommon in pure Python code. """ but to be honest I don't recall seeing this pattern commonly used. That's not to say that the feature might not be useful, but I think you'll need a bit more justification, and examples of code that would be improved by this proposal, if you want to get it accepted. Paul
On 09/13/2016 09:29 AM, Chris Angelico wrote:
On Wed, Sep 14, 2016 at 2:16 AM, Pim Schellart wrote:
A well-written PEP that looked very interesting.
But this seems to solve the problem nicely. In other words, you now need to explain why this particular class decorator is not good enough to solve the general problem. -- ~Ethan~
Hi all, On Tue, 13 Sep 2016 at 18:30 Chris Angelico <rosuav@gmail.com> wrote:
There's also this post from Guido a few years back in python-dev, which does something similar using metaclasses: https://mail.python.org/pipermail/python-dev/2008-January/076194.html I'm not sure it does exactly the same, but it is also an interesting approach. (Although I like the decorator syntax more.) The discussion that follows in that thread may also be of interest for this discussion. Ralph
On 14 September 2016 at 05:46, Ralph Broenink <ralph@ralphbroenink.net> wrote:
PEP 422 and its reference implementation had a more fleshed out implementation of that concept: https://www.python.org/dev/peps/pep-0422/#new-ways-of-using-classes With class namespaces becoming ordered by default, I ended up withdrawing PEP 422 in favour of the simpler PEP 487 that's actually in 3.6: https://www.python.org/dev/peps/pep-0487/ I do think Pim's proposal is an excellent exemplar of what a PEP should be, and if we *did* want to make class extension easier than it already is, then the suggested "continue class ..." syntax would be an elegant way of spelling it. However, I also believe the proposal founders on: 1. Class extensions are a fundamentally bad idea from a maintainability perspective, as they make the original class definition incomplete with no local indicator of its lack of completeness (hence the "don't do this" caveat on the relevant example in PEP 422) 2. There are already ways to do this using metaclasses that provide a local indicator at the point of definition that the affected class is extensible (i.e. the use of the custom metaclass, or inheritance from a base class that uses it) and hence that the given definition may be incomplete Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Dear All, Thank you for your feedback! While we think the decorator solution is not as writable, readable or discoverable as our proposed syntax we understand the desire to discourage a potential paradigm switch to `open classes’. We recognise that in many cases refactoring the code to eliminate the need for monkey patching is the preferable solution. However, there are many instances where this is not the case, such as the one Guido described in https://mail.python.org/pipermail/python-dev/2008-January/076194.html. Personally, we have encountered the need for this in many projects each of us have worked on. The fact that monkey patching is common enough lexicon to be mentioned in Python programming books highlights the fact that programmers may encounter this in codebases they work on. In particular when using third party libraries or code written in other languages wrapped into Python, the existing syntaxes are less readable and non-standard. Each project using such codes is thus forced to come up with their own custom solution leading to a somewhat fractured appearance in otherwise similar Python codebases. We had hoped to come up with a standardised solution. And if new syntax is not desirable, perhaps including other solutions such as the proposed decorator in the standard library is an option? Or do you feel this is a problem that should be solved by better documentation and understanding instead? Kind regards, Pim Schellart & Nate Lust
There shall be no standard syntax for monkey-patching. Decorators are fine. I recommend putting a monkey-patching package on PyPI and see if people use it. Maybe eventually it can be in the stdlib but I really don't want to encourage this idiom (even if it's sometimes useful). You should always consider other options (like petitioning upstream for a better API) before accepting the need to monkey-patch. On Wed, Sep 14, 2016 at 8:46 AM, Pim Schellart <p.schellart@princeton.edu> wrote:
-- --Guido van Rossum (python.org/~guido)
All, I must say It is still exciting to be at a point of getting an official declaration on a topic from Guido. We thank you for your time and feedback. It has been a good experience and hope to be able to contribute in some other way in the future. Nate and Pim On Wed, Sep 14, 2016 at 12:30 PM Guido van Rossum <guido@python.org> wrote:
Pim Schellart wrote:
This PEP proposes the introduction of new syntax to create a community standard, readable way to continue a definition for classes which are already defined.
-1. Monkeyatching is something that should be disccouraged rather than encouraged, since it makes code very hard to follow. -- Greg
Continuing the definition for a class feels to me like a strong anti-pattern for Python. If you absolutely must do this, I guess the presented decorator is available. It should be discouraged though, not be part of new syntax. I believe that if you find yourself doing this you should detect a bad choice smell, and think about expressing your goal with inheritance, delegation, or encapsulation instead. On Sep 13, 2016 9:17 AM, "Pim Schellart" <p.schellart@princeton.edu> wrote:
Hello developers, I have written a functional reference implementation for the proposed continue class syntax which can be found at https://raw.githubusercontent.com/natelust/continueClassCpythonPatch/master/... . This is a mercurial patch file created against python 3.5. An example of its functionality, and demonstration that it does not interfere with the existing continue keyword is as follows: # Define a class Foo, taking a message as an argument class Foo: def __init__(self, message): self.message = message # Continue the definition of the class, adding a method to print # the class message variable continue class Foo: def second(self): print(self.message) # Create an instance of the class inst = Foo('Hello World') # Demo the continued class inst.second() # Show that existing syntax still works for i in range(5): if i == 0: continue print(i) Thank you for your time and consideration, we will be happy to take any feedback or questions on the implementation. Nate Lust On Tue, Sep 13, 2016 at 12:17 PM Pim Schellart <p.schellart@princeton.edu> wrote:
On Wed, Sep 14, 2016 at 2:22 AM, nate lust <natelust@linux.com> wrote:
AIUI, the central core to this code is: + /* Iterate over the dictionary which was populated in the eval call. + Set an attribute in the class corresponding to the key, item pair + generated by the eval statement */ + dict_size = PyDict_Size(newdict); + dictkeys = PyDict_Keys(newdict); + for (i = 0; i < dict_size; i++ ){ + key = PyList_GetItem(dictkeys, i); + item = PyDict_GetItem(newdict, key); + PyObject_SetAttr(aclass, key, item); + } How does this handle special elements such as __module__, __dict__, and __doc__? Does it overwrite them? ChrisA
On Wed, Sep 14, 2016 at 2:16 AM, Pim Schellart <p.schellart@princeton.edu> wrote:
Did you know that you can actually abuse decorators to do this with existing syntax? Check out this collection of evil uses of decorators: https://github.com/Rosuav/Decorators/blob/master/evil.py I call them "evil" because they're potentially VERY confusing, but they're not necessarily bad. The monkeypatch decorator does basically what you're doing here, but with this syntax: @monkeypatch class A: x = 5 def foo(self): pass def bar(self): pass It's a little bit magical, in that it looks up the original class using globals(); this is partly deliberate, as it means you can't accidentally monkey-patch something from the built-ins, which will either fail, or (far worse) succeed and confuse everyone. ChrisA
On 13 September 2016 at 17:29, Chris Angelico <rosuav@gmail.com> wrote:
Also, it's worth noting that while Ruby has a tradition of "open" classes (as in, the ability to add methods to class definitions after those classes have been created) Python traditionally does not encourage such use. You claim in your proposal """ This pattern frequently occurs when extending classes from extension modules written in other languages, but is also not uncommon in pure Python code. """ but to be honest I don't recall seeing this pattern commonly used. That's not to say that the feature might not be useful, but I think you'll need a bit more justification, and examples of code that would be improved by this proposal, if you want to get it accepted. Paul
On 09/13/2016 09:29 AM, Chris Angelico wrote:
On Wed, Sep 14, 2016 at 2:16 AM, Pim Schellart wrote:
A well-written PEP that looked very interesting.
But this seems to solve the problem nicely. In other words, you now need to explain why this particular class decorator is not good enough to solve the general problem. -- ~Ethan~
Hi all, On Tue, 13 Sep 2016 at 18:30 Chris Angelico <rosuav@gmail.com> wrote:
There's also this post from Guido a few years back in python-dev, which does something similar using metaclasses: https://mail.python.org/pipermail/python-dev/2008-January/076194.html I'm not sure it does exactly the same, but it is also an interesting approach. (Although I like the decorator syntax more.) The discussion that follows in that thread may also be of interest for this discussion. Ralph
On 14 September 2016 at 05:46, Ralph Broenink <ralph@ralphbroenink.net> wrote:
PEP 422 and its reference implementation had a more fleshed out implementation of that concept: https://www.python.org/dev/peps/pep-0422/#new-ways-of-using-classes With class namespaces becoming ordered by default, I ended up withdrawing PEP 422 in favour of the simpler PEP 487 that's actually in 3.6: https://www.python.org/dev/peps/pep-0487/ I do think Pim's proposal is an excellent exemplar of what a PEP should be, and if we *did* want to make class extension easier than it already is, then the suggested "continue class ..." syntax would be an elegant way of spelling it. However, I also believe the proposal founders on: 1. Class extensions are a fundamentally bad idea from a maintainability perspective, as they make the original class definition incomplete with no local indicator of its lack of completeness (hence the "don't do this" caveat on the relevant example in PEP 422) 2. There are already ways to do this using metaclasses that provide a local indicator at the point of definition that the affected class is extensible (i.e. the use of the custom metaclass, or inheritance from a base class that uses it) and hence that the given definition may be incomplete Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Dear All, Thank you for your feedback! While we think the decorator solution is not as writable, readable or discoverable as our proposed syntax we understand the desire to discourage a potential paradigm switch to `open classes’. We recognise that in many cases refactoring the code to eliminate the need for monkey patching is the preferable solution. However, there are many instances where this is not the case, such as the one Guido described in https://mail.python.org/pipermail/python-dev/2008-January/076194.html. Personally, we have encountered the need for this in many projects each of us have worked on. The fact that monkey patching is common enough lexicon to be mentioned in Python programming books highlights the fact that programmers may encounter this in codebases they work on. In particular when using third party libraries or code written in other languages wrapped into Python, the existing syntaxes are less readable and non-standard. Each project using such codes is thus forced to come up with their own custom solution leading to a somewhat fractured appearance in otherwise similar Python codebases. We had hoped to come up with a standardised solution. And if new syntax is not desirable, perhaps including other solutions such as the proposed decorator in the standard library is an option? Or do you feel this is a problem that should be solved by better documentation and understanding instead? Kind regards, Pim Schellart & Nate Lust
There shall be no standard syntax for monkey-patching. Decorators are fine. I recommend putting a monkey-patching package on PyPI and see if people use it. Maybe eventually it can be in the stdlib but I really don't want to encourage this idiom (even if it's sometimes useful). You should always consider other options (like petitioning upstream for a better API) before accepting the need to monkey-patch. On Wed, Sep 14, 2016 at 8:46 AM, Pim Schellart <p.schellart@princeton.edu> wrote:
-- --Guido van Rossum (python.org/~guido)
All, I must say It is still exciting to be at a point of getting an official declaration on a topic from Guido. We thank you for your time and feedback. It has been a good experience and hope to be able to contribute in some other way in the future. Nate and Pim On Wed, Sep 14, 2016 at 12:30 PM Guido van Rossum <guido@python.org> wrote:
Pim Schellart wrote:
This PEP proposes the introduction of new syntax to create a community standard, readable way to continue a definition for classes which are already defined.
-1. Monkeyatching is something that should be disccouraged rather than encouraged, since it makes code very hard to follow. -- Greg
Continuing the definition for a class feels to me like a strong anti-pattern for Python. If you absolutely must do this, I guess the presented decorator is available. It should be discouraged though, not be part of new syntax. I believe that if you find yourself doing this you should detect a bad choice smell, and think about expressing your goal with inheritance, delegation, or encapsulation instead. On Sep 13, 2016 9:17 AM, "Pim Schellart" <p.schellart@princeton.edu> wrote:
participants (10)
-
Chris Angelico -
David Mertz -
Ethan Furman -
Greg Ewing -
Guido van Rossum -
nate lust -
Nick Coghlan -
Paul Moore -
Pim Schellart -
Ralph Broenink