[Twisted-Python] Phasing out old-style classes
On Thu Oct 8 20:08:12 EDT 2009, Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:
On Thu, Oct 8, 2009 at 7:59 PM, Mark Visser <markv at lumierevfx.com> wrote:
I've been bitten a couple times by twisted's use of old-style classes. Now that Jython is finally off the 2.2 branch, is there any real reason to stay backwards compatible?
Changing a class from old-style to new-style is an incompatible change. The difficulty is that if existing libraries use a particular class and inherit from it, changing the class to be new-style can have effects from changing how their descriptors work to causing an exception when their module is imported.
...
If old-style classes can be evolved into new-style classes while somehow following this policy, that would be great. The problem is that providing compatibility at this level is time-consuming and difficult. One problem in particular is that we don't want to litter the codebase with lots of "Foo" and "NewFoo" or "Foo2" sitting right next to it, so we would have to think of new names for everything.
I have some POC code for this. It provides a simple toggle at the start of the application to select between old style (the default) and new style classes. After that, a DeprecationWarning is issued every time an old style class is defined. The only user visible change is that old style classes gain an empty old-style base class. The idea being the old-style/new-style migration could be managed using the usual twisted deprecation plan. Thoughts?
On 02:39 pm, k.kelly.gordon@gmail.com wrote:
On Thu Oct 8 20:08:12 EDT 2009, Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:
On Thu, Oct 8, 2009 at 7:59 PM, Mark Visser <markv at lumierevfx.com> wrote:
I've been bitten a couple times by twisted's use of old-style classes. Now that Jython is finally off the 2.2 branch, is there any real reason to stay backwards compatible?
Changing a class from old-style to new-style is an incompatible change. The difficulty is that if existing libraries use a particular class and inherit from it, changing the class to be new-style can have effects from changing how their descriptors work to causing an exception when their module is imported. ...
If old-style classes can be evolved into new-style classes while somehow following this policy, that would be great. The problem is that providing compatibility at this level is time-consuming and difficult. One problem in particular is that we don't want to litter the codebase with lots of "Foo" and "NewFoo" or "Foo2" sitting right next to it, so we would have to think of new names for everything.
I have some POC code for this. It provides a simple toggle at the start of the application to select between old style (the default) and new style classes. After that, a DeprecationWarning is issued every time an old style class is defined. The only user visible change is that old style classes gain an empty old-style base class. The idea being the old-style/new-style migration could be managed using the usual twisted deprecation plan. Thoughts?
Process-wide switches are tricky. Some code may decide it wants new- style, while other code wants to stick with classic. They have a fight, one of them loses. I think that adding this is just an invitation for people to create more problems. I think that we should consider requests to make specific classes new- style (and grant them when doing so won't cause compatibility problems), make all new classes new-style, but otherwise leave this alone until 3.x is widely adopted. Jean-Paul
On Oct 24, 2009, at 10:50 AM, exarkun@twistedmatrix.com wrote:
I think that we should consider requests to make specific classes new- style (and grant them when doing so won't cause compatibility problems), make all new classes new-style, but otherwise leave this alone until 3.x is widely adopted.
While your argument makes sense to me, there's a fundamental problem with the way Python introduced new-style classes that creates an ongoing maintenance tension. I think we should start addressing the problem incrementally now (especially since it sounded like Kelly was volunteering for some work!) rather than put it off for one big chunk when we do a 3k migration. Let's say you're writing a new Twisted application today. You want to be prepared for the day when everything is new-style, but you also want to use existing Twisted functionality. You're using some long- standing library class, that looks like this: # in Twisted class Library: pass What do you do? Well, the obvious upgrade path here is to make a class which (A) inherits from "Library" to get Twisted functionality and (B) inherits from "object" to get new-style-ness. So you go ahead and write: # in your application class Application(object, Library): pass ... and that's great. It works, your class is new-style, and it gets all the library functionality that you want. Except now, you've made the old-style-ness of 'Library' a very important part of its interface. If, one day, we flip any kind of new-style switch, instead of neatly defining a new-style twisted-using class, your code will do this instead: Traceback (most recent call last): File "your-application.py", line 1, in <module> class Application(object, Library): pass TypeError: Error when calling the metaclass bases Cannot create a consistent method resolution order (MRO) for bases object, Library I would really like a more abstract declaration that applications can use in the meanwhile, to get new-style semantics but still allow inheritable classes to evolve. I suspect that we could do this with some kind of metaclass, but that seems ugly. More reasonable, I imagine, would be something like: class Application(newStyle(Library)): pass and we could implement 'newStyle' to do something sane in the face of either an old-style or a new-style "Library" class. Of course, there are still issues with behavior differences on decorators and so on, and so the same compatibility issues might still come up, but in my experience those issues are significantly less serious and easier to fix than "my entire program blows up and will not start unless I make all my classes old-style".
On Oct 25, 2009, at 1:38 AM, Glyph Lefkowitz wrote:
What do you do? Well, the obvious upgrade path here is to make a class which (A) inherits from "Library" to get Twisted functionality and (B) inherits from "object" to get new-style-ness. So you go ahead and write:
# in your application class Application(object, Library): pass
Maybe the answer "why not" is obvious and I should already know it, but wouldn't it be more obvious to write: class Application(Library, object): pass instead. That at least has the advantage of not blowing up if Library later becomes newstyle, right? James
On Oct 25, 2009, at 1:50 AM, James Y Knight wrote:
On Oct 25, 2009, at 1:38 AM, Glyph Lefkowitz wrote:
What do you do? Well, the obvious upgrade path here is to make a class which (A) inherits from "Library" to get Twisted functionality and (B) inherits from "object" to get new-style- ness. So you go ahead and write:
# in your application class Application(object, Library): pass
Maybe the answer "why not" is obvious and I should already know it, but wouldn't it be more obvious to write: class Application(Library, object): pass instead. That at least has the advantage of not blowing up if Library later becomes newstyle, right?
Huh. You may have just pointed out that I misunderstand the algorithm used to generate that error. I thought that certain other hierarchies in cases more complex than the trivial one I presented would also give you that error if something became new-style in the middle, but I can't seem to generate any hierarchies that cause a problem. Is it really this simple? Always put 'object' at the *end* of your bases- list to declare the intention "I want to be new-style" and you'll be OK? (If so, why is that?)
On Sun, Oct 25, 2009 at 5:32 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Oct 25, 2009, at 1:50 AM, James Y Knight wrote:
On Oct 25, 2009, at 1:38 AM, Glyph Lefkowitz wrote:
# in your application class Application(object, Library): pass
Maybe the answer "why not" is obvious and I should already know it, but wouldn't it be more obvious to write: class Application(Library, object): pass instead. That at least has the advantage of not blowing up if Library later becomes newstyle, right?
Huh. You may have just pointed out that I misunderstand the algorithm used to generate that error. I thought that certain other hierarchies in cases more complex than the trivial one I presented would also give you that error if something became new-style in the middle, but I can't seem to generate any hierarchies that cause a problem. Is it really this simple? Always put 'object' at the *end* of your bases-list to declare the intention "I want to be new-style" and you'll be OK? (If so, why is that?)
From my understanding of the C3 algorithm, the relative order of classes in a base class's mro will be preserved in classes that inherit from it. Since "object" inherits from nothing, it can always be shunted to the end of the mro, provided it hasn't ever been declared to be ahead of anything else.
On Sun, Oct 25, 2009 at 4:38 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Oct 24, 2009, at 10:50 AM, exarkun@twistedmatrix.com wrote:
I think that we should consider requests to make specific classes new- style (and grant them when doing so won't cause compatibility problems), make all new classes new-style, but otherwise leave this alone until 3.x is widely adopted.
While your argument makes sense to me, there's a fundamental problem with the way Python introduced new-style classes that creates an ongoing maintenance tension. I think we should start addressing the problem incrementally now (especially since it sounded like Kelly was volunteering for some work!) rather than put it off for one big chunk when we do a 3k migration.
Well yes I am. I am hoping that the discussion will get to a point where I understand what an acceptable solution might be, even if that should be like exarkun said "leave it alone/migrate classes one at a time".
I would really like a more abstract declaration that applications can use in the meanwhile, to get new-style semantics but still allow inheritable classes to evolve.
As noted by James, users of the Twisted library can add object to the end of their inheritance chain to get new style semantics for their classes. I was thinking more along the lines of being able to use new-style stuff inside the Twisted library.
On Sun, Oct 25, 2009 at 1:50 AM, <exarkun@twistedmatrix.com> wrote:
On 02:39 pm, k.kelly.gordon@gmail.com wrote:
On Thu Oct 8 20:08:12 EDT 2009, Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:
If old-style classes can be evolved into new-style classes while somehow following this policy, that would be great.
I have some POC code for this. It provides a simple toggle at the start of the application to select between old style (the default) and new style classes. After that, a DeprecationWarning is issued every time an old style class is defined.
Process-wide switches are tricky. Some code may decide it wants new-style, while other code wants to stick with classic. They have a fight, one of them loses. I think that adding this is just an invitation for people to create more problems.
I guess that would be a problem if an application used two libraries that use Twisted and each initialised Twisted differently. However, I was thinking this would be something that should only be called by the application (ie the thing with #! at the top) Roughly speaking - in the Twisted library: from twisted.object import ClassStyle class MustBeOldStyle(ClassStyle.old): pass class DoesntMatter(ClassStyle.default): pass In the application's top level file: #!/usr/bin/env python from twisted.object import ClassStyle ClassStyle.use_old_style() # or ClassStyle.use_new_style() Which would have the effect flipping ClassStyle.default between old and new style. Once set it can't be changed which is why it needs to be done before any Twisted classes are defined.
I think that we should consider requests to make specific classes new-style (and grant them when doing so won't cause compatibility problems), make all new classes new-style, but otherwise leave this alone until 3.x is widely adopted.
Jean-Paul
How could that be done in a way that was consistent with the deprecation policy? ie how could a warning be issued: "this class will be new style in a couple of releases", such that the developer could "fix" their code to remove the warning? Changing from old-style to new-style changes a number of class behaviours (notably: mro, __coerce__, assignment to Class.__dict__ and Class.__bases__), so its not really something that can be done silently to any class that might be inherited from. Presumably there will be a period when Twisted works on both 2.x and 3.x. At that point there will be two sets of users, the 3.x's that use only new-style classes and the 2.x's for which Twisted classes may be either old or new. It would probably be better to have switched to new-style before then and only have to support the use of new style classes.
On 08:38 am, k.kelly.gordon@gmail.com wrote:
On Sun, Oct 25, 2009 at 1:50 AM, <exarkun@twistedmatrix.com> wrote:
On 02:39 pm, k.kelly.gordon@gmail.com wrote:
On Thu Oct 8 20:08:12 EDT 2009, Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:
If old-style classes can be evolved into new-style classes while somehow following this policy, that would be great.
I have some POC code for this. It provides a simple toggle at the start of the application to select between old style (the default) and new style classes. After that, a DeprecationWarning is issued every time an old style class is defined.
Process-wide switches are tricky. Some code may decide it wants new-style, while other code wants to stick with classic. They have a fight, one of them loses. I think that adding this is just an invitation for people to create more problems.
I guess that would be a problem if an application used two libraries that use Twisted and each initialised Twisted differently. However, I was thinking this would be something that should only be called by the application (ie the thing with #! at the top)
Roughly speaking - in the Twisted library:
from twisted.object import ClassStyle
class MustBeOldStyle(ClassStyle.old): pass
class DoesntMatter(ClassStyle.default): pass
In the application's top level file:
#!/usr/bin/env python
from twisted.object import ClassStyle
ClassStyle.use_old_style() # or ClassStyle.use_new_style()
Which would have the effect flipping ClassStyle.default between old and new style. Once set it can't be changed which is why it needs to be done before any Twisted classes are defined.
I think that we should consider requests to make specific classes new-style (and grant them when doing so won't cause compatibility problems), make all new classes new-style, but otherwise leave this alone until 3.x is widely adopted.
Jean-Paul
How could that be done in a way that was consistent with the deprecation policy? ie how could a warning be issued: "this class will be new style in a couple of releases", such that the developer could "fix" their code to remove the warning?
Changing from old-style to new-style changes a number of class behaviours (notably: mro, __coerce__, assignment to Class.__dict__ and Class.__bases__), so its not really something that can be done silently to any class that might be inherited from.
Actually, I was thinking that we would only do it to classes that can not be inherited from. For example, no one's really supposed to be subclassing Deferred, so we might consider making Deferred new-style (perhaps for performance reasons, for example).
Presumably there will be a period when Twisted works on both 2.x and 3.x.
I think it is still early to presume this. :)
At that point there will be two sets of users, the 3.x's that use only new-style classes and the 2.x's for which Twisted classes may be either old or new. It would probably be better to have switched to new-style before then and only have to support the use of new style classes.
But, presuming this :) I don't think we need to have switched to new- style classes for support this scenario. We only have to have written all our code such that it continues to work even when there are no longer any classic classes. I think that the majority of code in Twisted already works even if you flip the new-style switch. We just need to identify and fix the few places that don't.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Oct 27, 2009, at 8:17 AM, exarkun@twistedmatrix.com wrote:
At that point there will be two sets of users, the 3.x's that use only new-style classes and the 2.x's for which Twisted classes may be either old or new. It would probably be better to have switched to new-style before then and only have to support the use of new style classes.
But, presuming this :) I don't think we need to have switched to new- style classes for support this scenario. We only have to have written all our code such that it continues to work even when there are no longer any classic classes. I think that the majority of code in Twisted already works even if you flip the new-style switch. We just need to identify and fix the few places that don't.
Is this something that can be handled by just running the tests with the right switch so we can see everything that passes without the switch but doesn't with? Is there a "new-style" switch or just -3? S
On Oct 27, 2009, at 10:32 AM, Steve Steiner (listsin) wrote:
Is this something that can be handled by just running the tests with the right switch so we can see everything that passes without the switch but doesn't with?
Is there a "new-style" switch or just -3?
Download: http://twistedmatrix.com/trac/browser/sandbox/foom/newstyler.py?format=txt And, before importing anything from twisted, run: import newstyler newstyler.newstyle_prefix('twisted') Or (perhaps easier), you could put those statements in your copy of twisted/__init__.py before it does anything else. I haven't run the tests to see what breaks; I'll leave that to you. :) James
participants (5)
-
exarkun＠twistedmatrix.com -
Glyph Lefkowitz -
James Y Knight -
Kelly -
Steve Steiner (listsin)