generic Liftable abc-mixin breaks at MRO
Dear all, I think I found a crucial usecase where the standard MRO does not work out. I would appreciate your help to still solve this usecase, or might MRO even be adapted? The idea is to build a generic Lift-type which I call this way because the derived classes should be able to easily lift from subclasses. So for example if I have an instance *a* from *class A* and a *class B(A)* I want to make *a* an instance of *B* in a straightforward way. My implementation (Python 2.7): import abc import inspect def use_as_needed(func, kwargs): meta = inspect.getargspec(func) if meta.keywords is not None: return meta(**kwargs) else: # not generic super-constructor - pick only the relevant subentries: return func(**{k:kwargs[k] for k in kwargs if k in meta.args}) class Liftable(object): __metaclass__ = abc.ABCMeta def __init__(self, **kwargs): use_as_needed(super(Liftable,self).__init__, kwargs) use_as_needed(self.__initialize__, kwargs) @abc.abstractmethod def __initialize__(self, **kwargs): return NotImplemented() class NoMatchingAncestor(RuntimeError): pass class NotLiftable(RuntimeError): pass def lift(self, new_class, **kwargs): # Stop Conditions: if self.__class__ is new_class: return # nothing to do elif new_class is object: # Base Case # break recursion at once: raise NoMatchingAncestor() elif new_class.__base__ is not Liftable: #to ensure this is save raise NotLiftable("Class {} is not Liftable (must be first parent)".format(new_class.__name__)) # recursive case: if not self.__class__ is new_class.__bases__[1]: lift(self, new_class.__bases__[1], **kwargs) # own case: self.__class__ = new_class use_as_needed(self.__initialize__, kwargs) and the example usecase: class A(object): def __init__(self, a): self.a = a class B(Liftable, A): def __initialize__(self, b): self.b = b a = A(1) print a.a, a.__class__ # 1 <class '__main__.A'> lift(a, B, b=2) print a.a, a.b, a.__class__ # 1 2 <class '__main__.B'> this works so far, however if I now put a further level of Liftable (which in principal already works with the generic definition class C(Liftable, B): def __initialize__(self, c): self.c = c I get the error TypeError: Error when calling the metaclass bases Cannot create a consistent method resolution order (MRO) for bases Liftable, B I read about MRO, and it seems to be the case that this setting somehow raises this generic Error, however I really think having such a Lifting is save and extremely useful - how can I make it work in python? (one further comment: switching the order of inheritance, i.e. class B(A, Liftable) will call A.__init__ before Liftable.__init__ which makes the whole idea senseless) Any constructive help is appreciated! best, Stephan
On 10.12.2015 09:58, Stephan Sahm wrote:
Why don't you use: a.__class__ = B ?
-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Dec 10 2015)
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/
On 10.12.2015 10:35, Stephan Sahm wrote:
Hmm, then I don't understand why you need a meta class for this. Could you explain the requirements that led up to needing a meta class ? If all you want to do is let the lift() know about a certain property of a class to do it's thing, an attribute or perhaps an empty base class would do the trick, e.g. class A: __liftable__ = True or perhaps: class A: def lift(self, ...): # lift operation goes here
-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Dec 10 2015)
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/
Thanks for asking, I am glad to explain the idea behind Liftable: My first implementation of *lift* just checked whether the class has an initialize method (and not being of type Liftable. However then B must look like so: class B(A): def __init__(self, b, a): super(B, self).__init__(a) self.__initialize__(b) def __initialize__(self, b): self.b = b this *__init__* method is generic for all Liftable-types in my sense, so I wanted to factorize it out, where the first guess I had was to use an abc-mixin. So one main point is that I want to make further initializations to my lifted instance which make it a truly valid instance of the new type. best, Stephan On 10 December 2015 at 10:44, M.-A. Lemburg <mal@egenix.com> wrote:
On Thu, Dec 10, 2015 at 09:58:37AM +0100, Stephan Sahm wrote:
I must admit I am not familiar with the terminology "Lift" in this context, so I may have missed something. But does this example help? py> class A(object): ... def spam(self): ... return "spam" ... py> class B(A): ... def spam(self): ... result = super(B, self).spam() ... return " ".join([result]*3) ... py> a = A() py> a.spam() 'spam' py> a.__class__ = B py> a.spam() 'spam spam spam' py> type(a) <class '__main__.B'> -- Steve
On Dec 10, 2015, at 00:58, Stephan Sahm <Stephan.Sahm@gmx.de> wrote:
Dear all,
I think I found a crucial usecase where the standard MRO does not work out. I would appreciate your help to still solve this usecase, or might MRO even be adapted?
First, do you have an actual use case for this? And are you really looking to suggest changes for Python 3.6, or looking for help with using Python 2.7 as-is? Anyway, I think the first problem here is that you're trying to put the same class, Liftable, on the MRO twice. That doesn't make sense--the whole point of superclass linearization is that each class only appears once in the list. If you weren't using a metaclass, you wouldn't see this error--but then you'd just get the more subtle problem that C can't be lifted from A to B because Liftable isn't getting called there. If you made Liftable a class factory, so two calls to Liftable() returned different class objects, then you might be able to make this work. (I suppose you could hide that from the user by giving Liftable a custom metaclass that constructs new class objects for each copy of Liftable in the bases list before calling through to type, but that seems like magic you really don't want to hide if you want anyone to be able to debut this code.) In fact, you could even make it Liftable(T), which makes your type Liftable from T, rather than from whatever class happens to come after you on the MRO chain. (Think about how this would work with other mixins, or pure-interface ABCs, or full multiple inheritance--you may end up declaring that C can be lifted from Sequence rather than B, which is nonsense, and which will be hard to debug if you don't understand the C3 algorithm.) Or, if you actually _want_ to be liftable from whatever happens to come next, then isn't liftability a property of the entire tree of classes, not of individual classes in that tree, so you should only be specifying Liftable once (either at A, or at B) in the hierarchy in the first place? From what I can tell, the only benefit you get from installing it twice is tricking the ABCMeta machinery into enforcing that all classes implement _initialize_ instead of just enforcing that one does; the easy solution there is to just write your own metaclass that does that check directly. Or maybe, instead of enforcing it, use it as a signal: build a "lift chain" for each Liftable type out of all classes on the MRO that directly implement _initialize_ (or just dynamically look for it as you walk the MRO in lift). So lift only works between those classes. I think that gets you all the same benefits as Liftable(T), without needing a class factory, and without having to specify it more than once on a hierarchy.
Dear Andrew, thank you very much for this impressively constructive response. It is for sure more constructive than I can react on now. For the concrete usecase, the Liftable-signal in might be the most interesting option, as then already the base class can inherit the Liftable-signal and can itself already use lift. However, I cannot see how to make the __init__ method conform in this setting, but by inidividual implementations (I in fact thought that enforcing it by the mixin makes things safer, and it of course should reduce boilerplate code) The Liftable(T) in fact seems also great, as I cannot see how to avoid this lifting from a false class in the Liftable-signal chain. I only want to Lift from one class at the moment, so this is in fact kind of what I was after. The rough outline would look like class B(Lift(A)): pass class C(Lift(B)): pass which seems rather beautiful to read - thank you very much for pointing this out. If you have an idea how to automatically create the right __init__ method when using the Liftable-signal-chain, I would highly welcome it. I myself need to recap some metaclass basics again before seriously tackling this. Best, Stephan On 10 December 2015 at 19:38, Andrew Barnert <abarnert@yahoo.com> wrote:
I now created both, and in fact the code for Liftable-signal seems much cleaner Nevertheless, the code is rather long for this thread, but it might be useful for someone import abc import inspect from contextlib import contextmanager def use_as_needed(func, kwargs): meta = inspect.getargspec(func) if meta.keywords is not None: return func(**kwargs) else: # not generic super-constructor - pick only the relevant subentries: return func(**{k:kwargs[k] for k in kwargs if k in meta.args}) class NotLiftable(RuntimeError): pass @contextmanager def super_liftable(cls, self): """ this is kind of a hack to replace super.super, however I haven't found any other nice way to do it """ if cls is object: raise NotLiftable() liftables = [l for l in cls.__bases__ if type(l).__name__ == "Liftable"] if not liftables: raise NotLiftable() orig_class = self.__class__ self.__class__ = liftables[0] yield self self.__class__ = orig_class def LiftableFrom(base_cls_name): class Liftable(type): def __init__(cls, name, bases, dct): # for base_cls nothing should be done, as this is the one to refer to by Lifting if not cls.__name__ == base_cls_name: if "__init__" in dct: raise TypeError("Descendents of Liftable are not allowed to have own __init__ method. Instead overwrite __initialize__") def lifted__init__(self, **kwargs): with super_liftable(cls, self) as s: use_as_needed(s.__init__, kwargs) if hasattr(self, "__initialize__"): use_as_needed(self.__initialize__, kwargs) cls.__init__ = lifted__init__ #setattr(cls, "__init__", lifted__init__) super(Liftable, cls).__init__(name, bases, dct) Liftable.base_cls_name = base_cls_name #Liftable.__name__ = "LiftableFrom" + base_cls_name # to show that this is possible return Liftable def lift(self, new_class, **kwargs): #TODO adapt to work with both definitions above # Stop Conditions: if self.__class__ is new_class: return # nothing to do elif new_class is object: # Base Case # break recursion at once: raise NotLiftable() ls = [l for l in new_class.__bases__ if type(l).__name__ == "Liftable"] if not ls: raise NotLiftable() # recursive case: if not self.__class__ is ls[0]: # it would also be possible to use tree like left-first-search here lift(self, ls[0], **kwargs) # own case: self.__class__ = new_class use_as_needed(self.__initialize__, kwargs) The least beautiful thing needed is too give the name of the base-class (in this case "A") as an additional parameter for the lift-meta-class. I haven't found a way to access A.__name__ directly or even better automatically get the class where this meta-class was originally inserted. A second point is that I had to use an own version of super(), however this works like a charm as far as I can see. The Metaclass ensures that any child must not have a __init__ but instead can only use __initialize__ like a replacement. Here an example which works: class A(object): __metaclass__ = LiftableFrom("A") def __init__(self, a): self.a = a class B(A): def __initialize__(self, b): print "initialize b" self.b = b class C(B): def __initialize__(self, c): print "initialize c" self.c = c a = A(a=1) a.a # 1 lift(a, C, b=2, c=3) print type(a) print a.a, a.b, a.c #initialize b #initialize c #<class '__main__.C'> #1 2 3 cheers, Stephan On 10 December 2015 at 22:05, Stephan Sahm <Stephan.Sahm@gmx.de> wrote:
On 10.12.2015 09:58, Stephan Sahm wrote:
Why don't you use: a.__class__ = B ?
-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Dec 10 2015)
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/
On 10.12.2015 10:35, Stephan Sahm wrote:
Hmm, then I don't understand why you need a meta class for this. Could you explain the requirements that led up to needing a meta class ? If all you want to do is let the lift() know about a certain property of a class to do it's thing, an attribute or perhaps an empty base class would do the trick, e.g. class A: __liftable__ = True or perhaps: class A: def lift(self, ...): # lift operation goes here
-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Dec 10 2015)
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/
Thanks for asking, I am glad to explain the idea behind Liftable: My first implementation of *lift* just checked whether the class has an initialize method (and not being of type Liftable. However then B must look like so: class B(A): def __init__(self, b, a): super(B, self).__init__(a) self.__initialize__(b) def __initialize__(self, b): self.b = b this *__init__* method is generic for all Liftable-types in my sense, so I wanted to factorize it out, where the first guess I had was to use an abc-mixin. So one main point is that I want to make further initializations to my lifted instance which make it a truly valid instance of the new type. best, Stephan On 10 December 2015 at 10:44, M.-A. Lemburg <mal@egenix.com> wrote:
On Thu, Dec 10, 2015 at 09:58:37AM +0100, Stephan Sahm wrote:
I must admit I am not familiar with the terminology "Lift" in this context, so I may have missed something. But does this example help? py> class A(object): ... def spam(self): ... return "spam" ... py> class B(A): ... def spam(self): ... result = super(B, self).spam() ... return " ".join([result]*3) ... py> a = A() py> a.spam() 'spam' py> a.__class__ = B py> a.spam() 'spam spam spam' py> type(a) <class '__main__.B'> -- Steve
On Dec 10, 2015, at 00:58, Stephan Sahm <Stephan.Sahm@gmx.de> wrote:
Dear all,
I think I found a crucial usecase where the standard MRO does not work out. I would appreciate your help to still solve this usecase, or might MRO even be adapted?
First, do you have an actual use case for this? And are you really looking to suggest changes for Python 3.6, or looking for help with using Python 2.7 as-is? Anyway, I think the first problem here is that you're trying to put the same class, Liftable, on the MRO twice. That doesn't make sense--the whole point of superclass linearization is that each class only appears once in the list. If you weren't using a metaclass, you wouldn't see this error--but then you'd just get the more subtle problem that C can't be lifted from A to B because Liftable isn't getting called there. If you made Liftable a class factory, so two calls to Liftable() returned different class objects, then you might be able to make this work. (I suppose you could hide that from the user by giving Liftable a custom metaclass that constructs new class objects for each copy of Liftable in the bases list before calling through to type, but that seems like magic you really don't want to hide if you want anyone to be able to debut this code.) In fact, you could even make it Liftable(T), which makes your type Liftable from T, rather than from whatever class happens to come after you on the MRO chain. (Think about how this would work with other mixins, or pure-interface ABCs, or full multiple inheritance--you may end up declaring that C can be lifted from Sequence rather than B, which is nonsense, and which will be hard to debug if you don't understand the C3 algorithm.) Or, if you actually _want_ to be liftable from whatever happens to come next, then isn't liftability a property of the entire tree of classes, not of individual classes in that tree, so you should only be specifying Liftable once (either at A, or at B) in the hierarchy in the first place? From what I can tell, the only benefit you get from installing it twice is tricking the ABCMeta machinery into enforcing that all classes implement _initialize_ instead of just enforcing that one does; the easy solution there is to just write your own metaclass that does that check directly. Or maybe, instead of enforcing it, use it as a signal: build a "lift chain" for each Liftable type out of all classes on the MRO that directly implement _initialize_ (or just dynamically look for it as you walk the MRO in lift). So lift only works between those classes. I think that gets you all the same benefits as Liftable(T), without needing a class factory, and without having to specify it more than once on a hierarchy.
Dear Andrew, thank you very much for this impressively constructive response. It is for sure more constructive than I can react on now. For the concrete usecase, the Liftable-signal in might be the most interesting option, as then already the base class can inherit the Liftable-signal and can itself already use lift. However, I cannot see how to make the __init__ method conform in this setting, but by inidividual implementations (I in fact thought that enforcing it by the mixin makes things safer, and it of course should reduce boilerplate code) The Liftable(T) in fact seems also great, as I cannot see how to avoid this lifting from a false class in the Liftable-signal chain. I only want to Lift from one class at the moment, so this is in fact kind of what I was after. The rough outline would look like class B(Lift(A)): pass class C(Lift(B)): pass which seems rather beautiful to read - thank you very much for pointing this out. If you have an idea how to automatically create the right __init__ method when using the Liftable-signal-chain, I would highly welcome it. I myself need to recap some metaclass basics again before seriously tackling this. Best, Stephan On 10 December 2015 at 19:38, Andrew Barnert <abarnert@yahoo.com> wrote:
I now created both, and in fact the code for Liftable-signal seems much cleaner Nevertheless, the code is rather long for this thread, but it might be useful for someone import abc import inspect from contextlib import contextmanager def use_as_needed(func, kwargs): meta = inspect.getargspec(func) if meta.keywords is not None: return func(**kwargs) else: # not generic super-constructor - pick only the relevant subentries: return func(**{k:kwargs[k] for k in kwargs if k in meta.args}) class NotLiftable(RuntimeError): pass @contextmanager def super_liftable(cls, self): """ this is kind of a hack to replace super.super, however I haven't found any other nice way to do it """ if cls is object: raise NotLiftable() liftables = [l for l in cls.__bases__ if type(l).__name__ == "Liftable"] if not liftables: raise NotLiftable() orig_class = self.__class__ self.__class__ = liftables[0] yield self self.__class__ = orig_class def LiftableFrom(base_cls_name): class Liftable(type): def __init__(cls, name, bases, dct): # for base_cls nothing should be done, as this is the one to refer to by Lifting if not cls.__name__ == base_cls_name: if "__init__" in dct: raise TypeError("Descendents of Liftable are not allowed to have own __init__ method. Instead overwrite __initialize__") def lifted__init__(self, **kwargs): with super_liftable(cls, self) as s: use_as_needed(s.__init__, kwargs) if hasattr(self, "__initialize__"): use_as_needed(self.__initialize__, kwargs) cls.__init__ = lifted__init__ #setattr(cls, "__init__", lifted__init__) super(Liftable, cls).__init__(name, bases, dct) Liftable.base_cls_name = base_cls_name #Liftable.__name__ = "LiftableFrom" + base_cls_name # to show that this is possible return Liftable def lift(self, new_class, **kwargs): #TODO adapt to work with both definitions above # Stop Conditions: if self.__class__ is new_class: return # nothing to do elif new_class is object: # Base Case # break recursion at once: raise NotLiftable() ls = [l for l in new_class.__bases__ if type(l).__name__ == "Liftable"] if not ls: raise NotLiftable() # recursive case: if not self.__class__ is ls[0]: # it would also be possible to use tree like left-first-search here lift(self, ls[0], **kwargs) # own case: self.__class__ = new_class use_as_needed(self.__initialize__, kwargs) The least beautiful thing needed is too give the name of the base-class (in this case "A") as an additional parameter for the lift-meta-class. I haven't found a way to access A.__name__ directly or even better automatically get the class where this meta-class was originally inserted. A second point is that I had to use an own version of super(), however this works like a charm as far as I can see. The Metaclass ensures that any child must not have a __init__ but instead can only use __initialize__ like a replacement. Here an example which works: class A(object): __metaclass__ = LiftableFrom("A") def __init__(self, a): self.a = a class B(A): def __initialize__(self, b): print "initialize b" self.b = b class C(B): def __initialize__(self, c): print "initialize c" self.c = c a = A(a=1) a.a # 1 lift(a, C, b=2, c=3) print type(a) print a.a, a.b, a.c #initialize b #initialize c #<class '__main__.C'> #1 2 3 cheers, Stephan On 10 December 2015 at 22:05, Stephan Sahm <Stephan.Sahm@gmx.de> wrote:
participants (4)
-
Andrew Barnert -
M.-A. Lemburg -
Stephan Sahm -
Steven D'Aprano