Sometimes you want to specific an optional method in an abstract base class. Currently we don't have a consistent way of doing so, instead having to mix in the old way of defining "abstract" methods: class MyABC(metaclass=ABCMeta): ... def do_something_optional(self): """An optional method for doing something.""" raise NotImplementedError This came up in issue 15502[1], where we are splitting importlib.abc.Finder into MetaPathFinder and PathEntryFinder. These have a method, invalidate_caches(), which is optional. It would be nice to have a new decorator akin to abstractmethod that would allow defining an optional interface in a consistent way. Something like optionalabstractmethod (or some better name). Then the above example would be like this: class MyABC(metaclass=ABCMeta): ... @optionalabstractmethod def do_something_optional(self): """An optional method for doing something.""" Thoughts? -eric [1] http://bugs.python.org/issue15502; Barry Warsaw voiced what I've considered on multiple occasions.
On Aug 02, 2012, at 09:26 AM, Eric Snow wrote:
Sometimes you want to specific an optional method in an abstract base class. Currently we don't have a consistent way of doing so, instead having to mix in the old way of defining "abstract" methods:
class MyABC(metaclass=ABCMeta): ...
def do_something_optional(self): """An optional method for doing something.""" raise NotImplementedError
This came up in issue 15502[1], where we are splitting importlib.abc.Finder into MetaPathFinder and PathEntryFinder. These have a method, invalidate_caches(), which is optional. It would be nice to have a new decorator akin to abstractmethod that would allow defining an optional interface in a consistent way. Something like optionalabstractmethod (or some better name). Then the above example would be like this:
class MyABC(metaclass=ABCMeta): ...
@optionalabstractmethod def do_something_optional(self): """An optional method for doing something."""
Thoughts?
+1 -Barry
Le 02/08/2012 17:26, Eric Snow a écrit :
Sometimes you want to specific an optional method in an abstract base class. Currently we don't have a consistent way of doing so, instead having to mix in the old way of defining "abstract" methods:
class MyABC(metaclass=ABCMeta): ...
def do_something_optional(self): """An optional method for doing something.""" raise NotImplementedError
Hi, What’s wrong with this? I think that the only thing making methods with @abstractmethod special is that they prevent instantiation if they are not overridden. If we remove that, the only remaining difference is terms of documentation. Maybe a new Sphinx directive could help? What would an optional abstract method do when it is called? Raise NotImplementedError? I’m not against this idea but I just don’t see how @optionalabstractmethod is better than raise NotImplementedError Regards, -- Simon Sapin
On Thu, Aug 2, 2012 at 9:37 AM, Simon Sapin <simon.sapin@kozea.fr> wrote:
Le 02/08/2012 17:26, Eric Snow a écrit :
Sometimes you want to specific an optional method in an abstract base class. Currently we don't have a consistent way of doing so, instead having to mix in the old way of defining "abstract" methods:
class MyABC(metaclass=ABCMeta): ...
def do_something_optional(self): """An optional method for doing something.""" raise NotImplementedError
Hi,
What’s wrong with this?
I think that the only thing making methods with @abstractmethod special is that they prevent instantiation if they are not overridden. If we remove that, the only remaining difference is terms of documentation. Maybe a new Sphinx directive could help?
What would an optional abstract method do when it is called? Raise NotImplementedError?
I’m not against this idea but I just don’t see how @optionalabstractmethod is better than raise NotImplementedError
Yeah, there isn't a huge win (particularly considering the implementation would tread on perilous ground*). The main motivation is consistency in the definition of abstract base classes. How would it work? Yeah, probably a lot like the NotImplementedError technique works. Or perhaps it would simply be removed by the ABCMeta when the class is built. -eric * If you have an hour or two to spare, manually trace trough the abc module and Objects/typeobject.c to see how abstract base classes work under the hood. Totally worth it!
On 2012-08-02, at 17:26 , Eric Snow wrote:
Sometimes you want to specific an optional method in an abstract base class. Currently we don't have a consistent way of doing so, instead having to mix in the old way of defining "abstract" methods:
class MyABC(metaclass=ABCMeta): ...
def do_something_optional(self): """An optional method for doing something.""" raise NotImplementedError
This came up in issue 15502[1], where we are splitting importlib.abc.Finder into MetaPathFinder and PathEntryFinder. These have a method, invalidate_caches(), which is optional. It would be nice to have a new decorator akin to abstractmethod that would allow defining an optional interface in a consistent way. Something like optionalabstractmethod (or some better name). Then the above example would be like this:
class MyABC(metaclass=ABCMeta): ...
@optionalabstractmethod def do_something_optional(self): """An optional method for doing something."""
Thoughts?
Wouldn't it be better to have a generic @optional applying to all @abstract*? e.g. class MyABC*metaclass=ABCMeta): @optional @abstractmethod def do_something_optional(self): """ an optional method for doing something """ (yeah I know all of them apart from @abstractmethod are deprecated, still it feels ugly and unreadable to have @optionalabstractmethod next to @abstractmethod)
On Thu, 2 Aug 2012 09:26:59 -0600 Eric Snow <ericsnowcurrently@gmail.com> wrote:
Sometimes you want to specific an optional method in an abstract base class. Currently we don't have a consistent way of doing so, instead having to mix in the old way of defining "abstract" methods:
class MyABC(metaclass=ABCMeta): ...
def do_something_optional(self): """An optional method for doing something.""" raise NotImplementedError
What's the problem with it exactly? Regards Antoine. -- Software development and contracting: http://pro.pitrou.net
On Fri, Aug 3, 2012 at 1:26 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Sometimes you want to specific an optional method in an abstract base class. Currently we don't have a consistent way of doing so, instead having to mix in the old way of defining "abstract" methods:
class MyABC(metaclass=ABCMeta): ...
def do_something_optional(self): """An optional method for doing something.""" raise NotImplementedError
Really, the phrase "optional abstract method" is a bit of an oxymoron. The point of an abstract method is that subclasses *must* implement it, either so that concrete method implementations provided by the base class can rely on it, or so that users of the class can rely on it. For interface methods that aren't mandatory, there are already a few different signalling methods currently in use within the language core and standard library: 1. Attribute checks Call method X if it exists, call method Y otherwise (e.g. the iterator protocol, which falls back to the sequence iterator if __iter__ doesn't exist, but __getitem__ does, and the backwards compatibility in the path entry finder protocol, which tries find_loader first, then falls back to find_module) 2. Setting attributes to None Call method X if it is not None (e.g. the hash protocol, where we added support for __hash__ = None to cope with the fact that object instances are hashable, but instances of subclasses that override __eq__ without also overriding __hash__ are not) 3. Implementing the method, but returning NotImplemented Call method if defined, treat a result of NotImplemented as if the method did not exist at all. (e.g. the binary operator protocols) 4. Throwing a particular exception (typically NotImplementedError) Call method if defined, treat the designated exception as if the method does not exist at all (e.g. optional methods in the IO stack) Now, the interesting point here is that these are all things that can't easily be defined in a way that is open to *programmatic* introspection. Here's how the four of them would currently look in a base class: # optional_method(appropriate_signature) can be defined by X instances # See the docs for details (users should handle the case where this method is not defined) # optional_method(appropriate_signature) can be defined by X instances # See the docs for details (users should handle the case where this method is set to None) optional_method = None def optional_method(appropriate_signature): """Details on the significance of optional_method Users should handle the case where this method returns NotImplemented """ return NotImplemented def optional_method(appropriate_signature): """Details on the significance of optional_method Users should handle the case where this method raises NotImplementedError """ raise NotImplementedError Expanding the ABC descriptor lexicon to accurately describe those 4 protocol variants (or at least some of them) in a base class may be worthwhile, but -1 on merely adding yet another variant. For example (this is rather ugly and I don't actually like it, but I wanted to illustrate the general point that the descriptor protocol allows all 4 cases to be distinguished): @optional(raise_on_instance_get=AttributeError) def optional_method(appropriate_signature): ... @optional(value_on_instance=None) def optional_method(appropriate_signature): ... @optional(result_on_method_call=NotImplemented) def optional_method(appropriate_signature): ... @optional(raise_on_method_call=NotImplementedError) def optional_method(appropriate_signature): ... Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Aug 2, 2012 at 10:45 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Now, the interesting point here is that these are all things that can't easily be defined in a way that is open to *programmatic* introspection.
I hadn't thought of it in these terms, but deep down this is exactly the itch that was making me squirm.
Expanding the ABC descriptor lexicon to accurately describe those 4 protocol variants (or at least some of them) in a base class may be worthwhile, but -1 on merely adding yet another variant.
Agreed on both counts. I think your example is along the right lines. To be honest, I'll probably let this idea ruminate a while, but you're explanation is meaningful right now. Thanks. -eric
On 8/3/12, Nick Coghlan <ncoghlan@gmail.com> wrote:
Really, the phrase "optional abstract method" is a bit of an oxymoron.
Only if you place too much faith in the details on the current implementation. One major use of an abstract class is to document an interface. The abstract class can only define (or even document, really, if you value docstrings) methods that it defines.
For interface methods that aren't mandatory, there are already a few different signalling methods currently in use within the language core and standard library:
1. Attribute checks Call method X if it exists, call method Y otherwise (e.g. the iterator protocol, which falls back to the sequence iterator if __iter__ doesn't exist, but __getitem__ does, and the backwards compatibility in the path entry finder protocol, which tries find_loader first, then falls back to find_module)
Assuming that another method will be called is a bit too restrictive, but this otherwise works well -- until you inherit from the abstract class, in which case the method will exist, even if it wasn't implemented. The least bad solution I've found is to use co-operative super, and write an identity function for the base class. Then if you need to know whether or not it was actually implemented, you can at least do an identity check on the method, if you're careful enough. -jJ
On 07/08/12 08:55, Jim Jewett wrote:
On 8/3/12, Nick Coghlan<ncoghlan@gmail.com> wrote:
Really, the phrase "optional abstract method" is a bit of an oxymoron.
Only if you place too much faith in the details on the current implementation.
I don't see that this has anything to do with implementation.
One major use of an abstract class is to document an interface. The abstract class can only define (or even document, really, if you value docstrings) methods that it defines.
I don't know that I accept that abstract classes are documentation. It seems to me that to be documentation, it has to be, well, documentation. Not a class hierarchy. The documentation might include the fact that something is an abstract class, but merely making something an abstract class is not in and of itself documentation. But even putting that aside, the interface that it (implicitly?) documents is surely *required* interface. If you can neglect to override an abstract method with impunity, then it shouldn't have been an abstract method in the first place. As I see it, "optional abstract method" means "You *must* implement this method, but if you don't, that's okay too". -- Steven
On Aug 7, 10:35 am, Steven D'Aprano <st...@pearwood.info> wrote:
I don't know that I accept that abstract classes are documentation. It seems to me that to be documentation, it has to be, well, documentation. Not a class hierarchy. The documentation might include the fact that something is an abstract class, but merely making something an abstract class is not in and of itself documentation.
Why not? What could I say here in textual documentation that isn't made obvious by declaring the interface? import abc class XBuilder(object): """Interface for setting up, updating & deleting objects for X""" __metaclass__ = abc.ABCMeta @abc.abstractmethod def setup(self): """Get the object ready for use""" @abc.abstractmethod def update(self): """Modify the object to reflect recent changes""" @abc.abstractmethod def delete(self): """Remove the object""" Good code _is_ documentation.
On Aug 7, 10:35 am, Steven D'Aprano <st...@pearwood.info> wrote:
But even putting that aside, the interface that it (implicitly?) documents is surely *required* interface. If you can neglect to override an abstract method with impunity, then it shouldn't have been an abstract method in the first place.
This I completely agree with. I don't understand the point of declaring an abstract method that you cannot guarantee will be on an implementation. Wouldn't it make more sense to define the optional aspect as a secondary interface? import abc class IStarter(object): __metaclass__ = abc.ABCMeta @abc.abstractmethod def start(self): """start it up""" class IStopper(object): __metaclass__ = abc.ABCMeta @abc.abstractmethod def stop(self): """shut it down""" class StartOnly(object): def start(self): print "starting!" class StartAndStop(object): def start(self): print "starting!!" def stop(self): print "stopping!!" IStarter.register(StartOnly) IStarter.register(StartAndStop) IStopper.register(StartAndStop)
On Tue, Aug 7, 2012 at 12:18 PM, alex23 <wuwei23@gmail.com> wrote:
On Aug 7, 10:35 am, Steven D'Aprano <st...@pearwood.info> wrote:
But even putting that aside, the interface that it (implicitly?) documents is surely *required* interface. If you can neglect to override an abstract method with impunity, then it shouldn't have been an abstract method in the first place.
This I completely agree with. I don't understand the point of declaring an abstract method that you cannot guarantee will be on an implementation.
Wouldn't it make more sense to define the optional aspect as a secondary interface?
No. That leads to a combinatorial explosion of interface classes that only makes sense in languages which don't readily support attribute level introspection (*cough*such-as-Java*cough*). Look at the IO stack for an example - think how much more complicated it would need to be if implementations weren't allowed to raise NotImplemented error for unsupported methods and each of those methods thus had an associated ABC. As I wrote in my earlier message, there are several ways to define a runtime protocol to check for optional methods in Python today. The problem is that *NONE* of them are particularly open to programmatic introspection, thus it is difficult for automatic documentation tools to flag them correctly the way they can flag abstract methods. To avoid people having to dig up the alternatives (all of which are used in the core or standard library in various situations): - the method/attribute may be missing entirely - the method/attribute is always present, but may be None - the method is always present, but returns NotImplemented by default - the method is always present, but raises NotImplementedError by default The first case you can't indicate in an ABC at all (except in a comment) The second case makes it difficult to add a docstring (although you can use a property to get around that) The latter two cases mean you have to actually *call* the method to find out if if is implemented or not So, here's a concrete suggestion that would be suitable for many cases where this is desired (although obviously not all such cases, due to backwards compatibility concerns, as well as cases where it is desirable that the base implementation be suitable for termination of a chain of cooperative super calls) 1. Add a __call__ definition to type(NotImplemented) that accepts arbitrary arguments and always raises NotImplementedError 2. Add an @optional decorator that functions roughly as follows class optional: def __init__(self, f): functools.update_wrapper(self, f) def __get__(self, *args): return NotImplemented A demo (using a custom "not implemented" marker and property rather than a custom descriptor):
def _not_implemented(*args, **kwds): ... raise NotImplementedError ... def optional(f): ... return property((lambda self: _not_implemented), doc=f.__doc__) ... class C: ... @optional ... def may_not_be_implemented(signature_details): ... """Documentation of this method""" ... C.may_not_be_implemented.__doc__ 'Documentation of this method' C.may_not_be_implemented <property object at 0x7fa5cd06afc8> C().may_not_be_implemented <function _not_implemented at 0x7fa5cd070e60> C().may_not_be_implemented is _not_implemented True C().may_not_be_implemented() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 2, in _not_implemented NotImplementedError
That specific approach does have the problem that you lose the signature details, so you'd probably want a custom descriptor that is recognised by inspect.getsignature() rather than reusing the property descriptor as I have done here. A custom descriptor would also be easier to pick out on the class object without needing an instance. Such an approach would improve the expressiveness of the ABC dialect, while remaining broadly consistent with current practices for marking optional methods. Most importantly, it would provide a fairly obvious way to flag optional methods in APIs in a way that is open to static introspection. However, it's not a universal solution. As noted above, it's not usable in any cases where you want the optional method to be usable as a terminal for multiple inheritance. It may be with some API tweaks, you could convert optional to a decorator factory that also handles the "suitable foundation for multiple inheritance" case. Perhaps the answer is even simpler than what I have above: perhaps the decorator could just set "f.__implemented__ = False", allowing introspection via "getattr(method, '__implemented__', True)". This is something that really needs to be explored before making a concrete proposal. Looking at the various ways optional methods are used in the standard library and interpreter core would be a good place to start. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Aug 6, 2012, at 10:02 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
No. That leads to a combinatorial explosion of interface classes that only makes sense in languages which don't readily support attribute level introspection (*cough*such-as-Java*cough*).
Yes, so don't do that. =) What's the matter with having a set of *distinct* interface classes and having a given *implementation* cherry pick the appropriate ones?
On Tue, Aug 7, 2012 at 4:51 PM, James William Pye <x@jwp.io> wrote:
On Aug 6, 2012, at 10:02 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
No. That leads to a combinatorial explosion of interface classes that only makes sense in languages which don't readily support attribute level introspection (*cough*such-as-Java*cough*).
Yes, so don't do that. =)
What's the matter with having a set of *distinct* interface classes and having a given *implementation* cherry pick the appropriate ones?
If the additional ABCs are defined with an appropriate fallback to a method existence check (ala collections.Hashable) then it may be OK. However, the long "implements" lists that Java is prone to is precisely the boilerplate I consider unacceptable (and antithetical to ducktyping). The point of this discussion is that there really isn't an obvious general purpose mechanism that easily lets you document the existence of an optional method in an ABC (such that tools like pydoc will see it), without inadvertently making it look like that method is implemented. The "x.__hash__ is None" trick that is used to get around object.__hash__ existing by default certainly works, but is definitely not the typical behaviour. Other approaches, like returning NotImplemented or raising NotImplementedError have the problem that the only way to find out if they're implemented or not is to call them, which may have unwanted side effects if they *are* implemented. That's why the IO stack has methods like "seekable()" - to ask the question "is the seek() method implemented?". Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Aug 7, 2012, at 12:42 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
However, the long "implements" lists that Java is prone to is precisely the boilerplate I consider unacceptable (and antithetical to ducktyping).
*shrug* I figure if you run into a case where you see a long combination in common use, go ahead and make another ABC consisting of that combination. Or just throw them in a sequence and map a methodcaller.
there really isn't an obvious general purpose mechanism that easily lets you document the existence of an optional method in an ABC
ISTM the proposed mechanism is a bit of a contradiction, and the value provided to documentation extractors does not appear to be particularly significant. However, annotations on the implementation may be a reasonable solution for documentation extractors: def foo() -> NotImplemented: ...
That's why the IO stack has methods like "seekable()" - to ask the question "is the seek() method implemented?".
Well, hopefully for ESPIPE…Which doesn't appear to be the case.. =(
On 8/6/12, Steven D'Aprano <steve@pearwood.info> wrote:
On 07/08/12 08:55, Jim Jewett wrote:
One major use of an abstract class is to document an interface. The abstract class can only define (or even document, really, if you value docstrings) methods that it defines.
I don't know that I accept that abstract classes are documentation. It seems to me that to be documentation, it has to be, well, documentation.
If the code is short and seems to be clear, many people will never look at the external documentation. In that case, leaving surprises out of the docstring is almost as bad as not documenting them at all.
... merely making something an abstract class is not in and of itself documentation.
Agreed, but that abstract class itself should clearly document the contract that concrete implementations (and their users) must follow.
But even putting that aside, the interface that it (implicitly?) documents is surely *required* interface.
Separating, for example, pen.draw from gunfighter.draw was one of the primary use cases of abstract base classes. _abcoll.py is dealing with infrastructure where python already defines the double-underscore methods; for now, pretend it were an ordinary domain module, using normal unreserved names. A Container need not implement __len__. Since we're pretending that the name wouldn't already be reserved by python, a concrete Container could use the name for some domain-specific attribute, such as LoudEngineNoise. Every specialized container in the module (Sequence, Set, Mapping, even MappingView) does need to have a __len__ method (ruling out some factory objects). They even all define it with the name meaning, by also inheriting from Sized. I would prefer to see the name reserved and defined at the Container level. It would still be OK to create a Container that could not tell you its length, but it would not be OK to create a Container using __len__ for some other purpose. For reasons that are no longer obvious, a MappingView needs to be Sized, but does not need to be an Iterable or a Container. I haven't yet seen a use for MappingView except to group the derived classes KeysView, ItemsView, and ValuesView -- all of which are Iterable Containers. KeysView and ItemsView also contain identical _from_iterable methods.* Reusing the names associated with Iterable or Container (remember we're pretending they weren't reserved by the double-underscore) for some other use would be very confusing. So would reusing _from_iterable, if it weren't semi-private. MappingView should be able to reserve these names and forbid such unrelated use. I'm not sure it needs to actually enforce the prohibition, but it should be able to at least express the prohibition in a manner visible to introspection. * Well, identical code. I'm assuming that the expected input type differs. -jJ
participants (10)
-
alex23 -
Antoine Pitrou -
Barry Warsaw -
Eric Snow -
James William Pye -
Jim Jewett -
Masklinn -
Nick Coghlan -
Simon Sapin -
Steven D'Aprano