Fwd: Consider allowing the use of abstractmethod without metaclasses
Hi, Neil.
I'm +1 with your idea in performance point of view. Some people having other language background (C# or Java) want to use ABC like Java's interface. But ABC is too heavy to use only for checking abstract methods. It uses three inefficient WeakSet [1] and it overrides isinstance and issubclass with slow Python implementation. [1] WeakSet is implemented in Python, having one __dict__, list and two sets. And C implementation of gathering abstract methods will reduce Python startup time too. Because `_collections_abc` module has many ABCs and `os` module import it. ## in abc.py # Helper function. ABCMeta use this too. # And Python 3.7 can have C implementation of this. def _init_abstractclass(cls, bases=None): # Compute set of abstract method names if bases is None: bases = cls.__bases__ abstracts = {name for name, value in vars(cls).items() if getattr(value, "__isabstractmethod__", False)} for base in bases: for name in getattr(base, "__abstractmethods__", set()): value = getattr(cls, name, None) if getattr(value, "__isabstractmethod__", False): abstracts.add(name) cls.__abstractmethods__ = frozenset(abstracts) class Abstract: __init_subclass__ = _init_abstractclass ## usage import abc class AbstractBar(abc.Abstract): @abc.abstractmethod def bar(self): ... Bests, INADA Naoki <songofacandy@gmail.com>
Le 20 juil. 2017 3:49 AM, "INADA Naoki" <songofacandy@gmail.com> a écrit :
I'm +1 with your idea in performance point of view.
(...) But ABC is too heavy to use only for checking abstract methods. It uses three inefficient WeakSet [1] and it overrides isinstance and issubclass with slow Python implementation. I don't think that we can get ride of abc from the io and importlib. They are here to stay. I hear your performance analysis. Why not making abc faster instead of trying to workaround abc for perf issue? Someone already wrote WeakrefSet, a PR is waiting for our review! Victor
Hi, Victor.
Why not making abc faster instead of trying to workaround abc for perf issue?
Current ABC provides: a) Prohibit instantiating without implement abstract methods. b) registry based subclassing People want Java's interface only wants (a). (b) is unwanted side effect. Additionally, even if CPython provide C implementation of ABCMeta, other Python implementations won't. So Abstract Class (not ABC) may be nice on such implementations too. I'm +1 to implement abc module in C. And I think (a) can be nice first step, instead of implement all at once. Regards, INADA Naoki <songofacandy@gmail.com>
Good discussion so far. Please let me know if I can help with implementation or documentation. On Thu, Jul 20, 2017 at 8:40 AM INADA Naoki <songofacandy@gmail.com> wrote:
Right. (b) is only unwanted because it requires metaclasses, and metaclasses put constraints on inheritance. I have switched to using the "AbstractBaseClass" class I defined above in my own code. If that is the sort of solution that will be undertaken, then there is a question of what to call this class. If instead, every class will get this functionality automatically (which is more elegant), then someone needs to show that performance is unaffected. Also, this whole thing might not be that important if (as Guido implies) linters supplant this functionality. Although, linters would not catch the rare case where classes are programatically-generated.
To be honest, I am not very happy with addition of a new special class. Imagine that the PEP 544 will be accepted (and I hope so). Then we would have, abstract classes, abstract base classes, and protocols. I think users will be overwhelmed by having three similar concepts instead of one. I think we still could squeeze a lot of performance from good old ABCs by optimizing various parts and reimplementing some parts in C. In fact, my desire to optimize and rewrite ABCMeta in C is partially due to reluctance to add yet another concept of "abstractness". -- Ivan
On Fri, Jul 21, 2017 at 12:12 AM, Ivan Levkivskyi <levkivskyi@gmail.com> wrote:
Hmm, couldn't split protocol and ABC? Of course, existing ABCs should be ABC for backward compatibility. But any reason to force using ABCMeta for user defined classes? I hate subclassing ABC because concrete classes which mix-in some ABC are forced to use it.
Even if it's implemented in C, issubclass implementation is much complicated than normal type. I don't want to introduce unnecessary complexity because I'm minimalist. Regards,
On 20 July 2017 at 19:51, INADA Naoki <songofacandy@gmail.com> wrote:
Unfortunately no, it was considered and rejected for various reasons (in particular to provide smooth transition to protocols).
This complexity is already there, and attempt to reduce might lead to actually an increase of complexity. This is probably the case where I would be with Raymond in terms of performance vs ease of maintenance. -- Ivan
INADA Naoki <songofacandy@gmail.com> On Fri, Jul 21, 2017 at 2:59 AM, Ivan Levkivskyi <levkivskyi@gmail.com> wrote:
Sorry about my poor English. "split" meant "optionally ABC". I understand that existing classes (like typing.Sequence) must be ABC. But why new user defined protocol must be ABC?
Sorry again. I meant I don't want import the complexity to my class when I don't need it. In other words, I hate inheriting ABC when I don't need register based subclass.
-- Ivan
On Jul 20, 2017 05:39, "INADA Naoki" <songofacandy@gmail.com> wrote: Hi, Victor.
Why not making abc faster instead of trying to workaround abc for perf issue?
Current ABC provides: a) Prohibit instantiating without implement abstract methods. b) registry based subclassing People want Java's interface only wants (a). (b) is unwanted side effect. Except (b) is what allows you to subclass an ABC without using the ABC metaclass :-) I wonder if it would make sense to go further and merge *both* of these features into regular classes. Checking for @abstractmethod in type.__new__ surely can't be that expensive, can it? And if regular types supported 'register', then it would allow for a potentially simpler and faster implementation. Right now, superclass.register(subclass) has to work by mutating superclass, because that's the special ABCMeta object, and this leads to complicated stuff with weakrefs and all that. But if this kind of nominal inheritance was a basic feature of 'type' itself, then it could work by doing something like subclass.__nominal_bases__ += (superclass,) and then precalculating the "nominal mro" just like it already precalculates the mro, so issubclass/isinstance would remain fast. I guess enabling this across the board might cause problems for C classes whose users currently use isinstance to get information about the internal memory layout. -n
But it affects startup time. It iterate all of the namespace and tries `getattr(obj, `__isabstractmethod__`, False). It means massive AttributeErrors are raised and cleared while loading large library. OTOH, I have another idea: class AbstractFoo: def foo(self): ... __abstractmethods__ = ("foo",) In this idea, `type.__new__` can check only `__abstractmethods__`.
I don't like it. In 99.9% of my classes, I don't need register based subclassing.
Le 20 juil. 2017 3:49 AM, "INADA Naoki" <songofacandy@gmail.com> a écrit :
I'm +1 with your idea in performance point of view.
(...) But ABC is too heavy to use only for checking abstract methods. It uses three inefficient WeakSet [1] and it overrides isinstance and issubclass with slow Python implementation. I don't think that we can get ride of abc from the io and importlib. They are here to stay. I hear your performance analysis. Why not making abc faster instead of trying to workaround abc for perf issue? Someone already wrote WeakrefSet, a PR is waiting for our review! Victor
Hi, Victor.
Why not making abc faster instead of trying to workaround abc for perf issue?
Current ABC provides: a) Prohibit instantiating without implement abstract methods. b) registry based subclassing People want Java's interface only wants (a). (b) is unwanted side effect. Additionally, even if CPython provide C implementation of ABCMeta, other Python implementations won't. So Abstract Class (not ABC) may be nice on such implementations too. I'm +1 to implement abc module in C. And I think (a) can be nice first step, instead of implement all at once. Regards, INADA Naoki <songofacandy@gmail.com>
Good discussion so far. Please let me know if I can help with implementation or documentation. On Thu, Jul 20, 2017 at 8:40 AM INADA Naoki <songofacandy@gmail.com> wrote:
Right. (b) is only unwanted because it requires metaclasses, and metaclasses put constraints on inheritance. I have switched to using the "AbstractBaseClass" class I defined above in my own code. If that is the sort of solution that will be undertaken, then there is a question of what to call this class. If instead, every class will get this functionality automatically (which is more elegant), then someone needs to show that performance is unaffected. Also, this whole thing might not be that important if (as Guido implies) linters supplant this functionality. Although, linters would not catch the rare case where classes are programatically-generated.
To be honest, I am not very happy with addition of a new special class. Imagine that the PEP 544 will be accepted (and I hope so). Then we would have, abstract classes, abstract base classes, and protocols. I think users will be overwhelmed by having three similar concepts instead of one. I think we still could squeeze a lot of performance from good old ABCs by optimizing various parts and reimplementing some parts in C. In fact, my desire to optimize and rewrite ABCMeta in C is partially due to reluctance to add yet another concept of "abstractness". -- Ivan
On Fri, Jul 21, 2017 at 12:12 AM, Ivan Levkivskyi <levkivskyi@gmail.com> wrote:
Hmm, couldn't split protocol and ABC? Of course, existing ABCs should be ABC for backward compatibility. But any reason to force using ABCMeta for user defined classes? I hate subclassing ABC because concrete classes which mix-in some ABC are forced to use it.
Even if it's implemented in C, issubclass implementation is much complicated than normal type. I don't want to introduce unnecessary complexity because I'm minimalist. Regards,
On 20 July 2017 at 19:51, INADA Naoki <songofacandy@gmail.com> wrote:
Unfortunately no, it was considered and rejected for various reasons (in particular to provide smooth transition to protocols).
This complexity is already there, and attempt to reduce might lead to actually an increase of complexity. This is probably the case where I would be with Raymond in terms of performance vs ease of maintenance. -- Ivan
INADA Naoki <songofacandy@gmail.com> On Fri, Jul 21, 2017 at 2:59 AM, Ivan Levkivskyi <levkivskyi@gmail.com> wrote:
Sorry about my poor English. "split" meant "optionally ABC". I understand that existing classes (like typing.Sequence) must be ABC. But why new user defined protocol must be ABC?
Sorry again. I meant I don't want import the complexity to my class when I don't need it. In other words, I hate inheriting ABC when I don't need register based subclass.
-- Ivan
On Jul 20, 2017 05:39, "INADA Naoki" <songofacandy@gmail.com> wrote: Hi, Victor.
Why not making abc faster instead of trying to workaround abc for perf issue?
Current ABC provides: a) Prohibit instantiating without implement abstract methods. b) registry based subclassing People want Java's interface only wants (a). (b) is unwanted side effect. Except (b) is what allows you to subclass an ABC without using the ABC metaclass :-) I wonder if it would make sense to go further and merge *both* of these features into regular classes. Checking for @abstractmethod in type.__new__ surely can't be that expensive, can it? And if regular types supported 'register', then it would allow for a potentially simpler and faster implementation. Right now, superclass.register(subclass) has to work by mutating superclass, because that's the special ABCMeta object, and this leads to complicated stuff with weakrefs and all that. But if this kind of nominal inheritance was a basic feature of 'type' itself, then it could work by doing something like subclass.__nominal_bases__ += (superclass,) and then precalculating the "nominal mro" just like it already precalculates the mro, so issubclass/isinstance would remain fast. I guess enabling this across the board might cause problems for C classes whose users currently use isinstance to get information about the internal memory layout. -n
But it affects startup time. It iterate all of the namespace and tries `getattr(obj, `__isabstractmethod__`, False). It means massive AttributeErrors are raised and cleared while loading large library. OTOH, I have another idea: class AbstractFoo: def foo(self): ... __abstractmethods__ = ("foo",) In this idea, `type.__new__` can check only `__abstractmethods__`.
I don't like it. In 99.9% of my classes, I don't need register based subclassing.
participants (5)
-
INADA Naoki -
Ivan Levkivskyi -
Nathaniel Smith -
Neil Girdhar -
Victor Stinner