check interfaces, isducktype(X, A)
Hi, I've made a try of a function that check object/class members for duck typing. For now I've basically called it isducktype(X, A) it returns true if: - X has all attributes of A - X methods have same number of arguments than corresponding A methods or if A.__ducktypecheck__(X) returns True Behaviour looks like isinstance and issubclass (support tuples...). To test it: The patch with example is here: https://gist.github.com/apieum/7751279 Or you can use my branch here: https://bitbucket.org/gsalvan/cpython/commits/branch/default Any opinion about a feature like this ?
Sounds like a cute thing to post on PyPI and see how popular it becomes. You have until the 3.5 release cycle starts to gather votes and improvements. There are lots of edge cases I could see going wrong, so I'm kind of skeptical of having this in the stdlib without strong caveats about its usefulness. Maybe it could go in the inspect module, which is traditionally full of heuristics. On Mon, Dec 2, 2013 at 7:55 AM, Gregory Salvan <apieum@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
Hi! On Mon, Dec 02, 2013 at 04:55:22PM +0100, Gregory Salvan <apieum@gmail.com> wrote:
Isn't the requirement too strong? When I need a file-like object I seldom need more than .read() and .write() methods, and even of those I seldom need both at once -- I usually need one or the other. Testing for strict conformance is not duck typing IMO -- it's interface. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Tue, Dec 3, 2013 at 3:11 AM, Oleg Broytman <phd@phdru.name> wrote:
But you could have a "ReadableFile" that has a .read() method and whatever else you need, thus using that class as a sort of interface. With a little careful scripting and introspection, you might even be able to craft that by source code analysis, which would be extremely cool (eg you run this script over your source code and it notes that elements of this list are indexed with these five keywords, so it constructs a class with those five so you can check what goes into the list). Whether it's actually _useful_ or not remains to be seen, but it would certainly be cool. ChrisA
Ok for pypi. (I've never tried to make python C extension, I feel it's going to be epic :) ) @Oleg you can override default behaviour with __ducktypecheck__ it's not as strict as an interface as var names can be differents and can have defaults. then instead of declaring an interface with "read" and "write", you just create a base class with these methods. What solution do you suggest ? I've made a function that returns public members only, maybe this method can be exposed in the api too. 2013/12/2 Oleg Broytman <phd@phdru.name>
On Tue, Dec 3, 2013 at 3:33 AM, Gregory Salvan <apieum@gmail.com> wrote:
Ok for pypi. (I've never tried to make python C extension, I feel it's going to be epic :)
Is there any reason this can't be implemented in pure Python? It'd be a lot easier to figure things out that way. Maybe a C implementation could come later, but for a first cut, this ought to be possible in Python. ChrisA
On Mon, Dec 02, 2013 at 05:33:10PM +0100, Gregory Salvan <apieum@gmail.com> wrote:
I don't suggest any solution as I don't have a problem to solve with interfaces. You are going to implement another interface checker in addition to existing ones? https://pypi.python.org/pypi?%3Aaction=search&term=interface&submit=search No objections from me. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Mon, Dec 02, 2013 at 08:04:19PM +0200, ?????????? <elazarg@gmail.com> wrote:
I'm sure the terminology in Python is quite stable. There are interfaces (there are many implementations, there is one rejected PEP and one deferred PEP) and there is duck typing. Formalizing duck typing is IMO constructing an interface. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
02.12.13 17:55, Gregory Salvan написав(ла):
class FooReader: def read(self, n=None): ... class BarReader: def read(self, amount=-1): ... class BazReader: def read(self, size=-1, chars=-1, firstline=False): ... Instances of all three classes can be used where needed an object with the read() method which are called with one positional argument or without arguments. Is it possible to use isducktype() for such case?
@Serhiy Not really (only with FooReader, BarReader) as it checks if functions co_argcount and co_kwonlyargcount are equals. In this case it requires BazReader.__ducktypecheck__ to be implemented. I had a doubt when doing this, I agree with you isducktype(BazReader, FooReader) can return True (whereas isducktype(FooReader, BazReader) should stay false) This can easily be changed. @Oleg: The fact there is a lot of interface implementations means certainly it's something needed. Each implementation I've seen add burden on objects whereas the only thing personnally I need, is to be sure I'll be able to call some methods off injected objects. @spir: Names can easily be checked, but I can't see why doing this, it add constraints to developpers without adding safety. @Andrew +1 Maybe inspect is more appropriate as it provides ArgSpec and getmembers ? My code is crappy, I've just do it like that to train me in C then tought it might be interesting to get feedback. 2013/12/2 Serhiy Storchaka <storchaka@gmail.com>
On Dec 2, 2013, at 17:21, Gregory Salvan <apieum@gmail.com> wrote:
@Andrew +1 Maybe inspect is more appropriate as it provides ArgSpec and getmembers ?
Sure, using inspect for the implementation _also_ makes sense. But my point is more about the interface to your new thing. Your thing checks most of the same stuff that ABCs check, but maybe not all of it, and also adds checking for compatible argspecs. (I believe that's the only new feature it adds; am I right?) Meanwhile, it uses a new isducktype function instead of working with normal issubclass/isinstance, and a completely different way of registering explicit compliance, and so on. Instead, why not just extend ABCMeta with a new subclass that just adds the check for compatible argspecs (maybe using inspect, as you say) and takes advantage of all the other stuff ABC already does? Besides being less code, which isn't that big of a deal, it's less of a radical conceptual change, which is a bigger deal, and it means that all kinds of code that relies on isinstance today can work with no changes.
On 3 December 2013 04:42, Andrew Barnert <abarnert@yahoo.com> wrote:
Shouldn't this be tied to ABCs rather than redesigning and reimplementing most of ABC to add a little bit on top?
This seems like an apropos place for this link: http://docs.python.org/3/library/abc#abc.ABCMeta.__subclasshook__ That's the existing "formalised ducktyping" hook, that already allows things like the following:
The advantage ABCs have over ducktyping alone is that subclassing and explicit registration allow ambiguities like the one between the Sequence and Mapping interfaces to be resolved (you can't always just ducktype those, as they have the same methods - they differ only in how the __*item__ methods handle slice objects). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Sorry Andrew, I'm not sure to understand all your point. I don't believe it reimplements things of ABC, or I don't see what. I agree ABC solution is the cleanest, whereas I would be able to check functions, stay on a runtime checking and being free to define or not a kind of protocol. Before reading your comments on ABC, I've released the python version of "isducktype": https://github.com/apieum/ducktype I've modified some behaviours within your comments and made it more stable. It would be interesting to have the version wich use ABCMeta, so I'm going to implement it. I've thought to name it SignedMeta, or ProtocolMeta, any opinion ? 2013/12/3 Nick Coghlan <ncoghlan@gmail.com>
On Dec 3, 2013, at 21:22, Gregory Salvan <apieum@gmail.com> wrote:
Sorry Andrew, I'm not sure to understand all your point. I don't believe it reimplements things of ABC, or I don't see what.
Almost everything you did reimplements features of ABCs. The builtin isinstance and issubclass functions can check whether you implement the methods of a type, and you can override that by explicitly declaring that you implement the type. That's the core feature set of your proposal, and of ABCs; you've just implemented the same thing with a different API, plus and minus a few minor details. If you haven't read PEP 3119, you really should. (It's also worth reading rejected predecessors like 245/246 and successors like 3124, and of course the pre-existing third party stuff like PyProtocols and the Interface types in Twisted and Zope.)
I agree ABC solution is the cleanest, whereas I would be able to check functions, stay on a runtime checking and being free to define or not a kind of protocol.
I don't understand what you mean here. ABCs can check methods. They do so at runtime. You're free to define or not define any protocol you want. What you have to add is checking the argspecs of the methods rather than just their names. That's a great idea, but it's a much better idea as an addition to what's already there than as a complete parallel API to very similar functionality. And if this were added to the stdlib, I'm pretty sure it would be a modification/extension of the abc module, not a competing and similar but not identical module.
First, it sounds like you're focusing on the wrong part of the name--ABCMeta is a metaclass for implementing ABCs; leaving out the ABC part sounds like you're designing a metaclass for something completely unrelated. Less seriously, ProtocolMeta sounds like it's going to add more PyProtocols-style functionality (in particular, adapters from one protocol to another), but there's only so many synonyms available and they're all taken, so I don't think that's a major problem. As for SignedMeta, I don't even know what that's meant to imply. Is it to do with method signatures? If so, I think "signature" is a better word; you don't really think about anything being "signed" in a method signature; that's dragging in associations from a different meaning of the word. But I don't have any better suggestions. Protocol, interface, prototype, type, abstract type, abstract base... As I said, all of the names have already been used up, repeatedly, and none of them have any particular connotations that would explain how your module, metaclass, function, etc. differ from abc. In fact, I'd probably just come up with a stupid package name like abc2 and stick in modules named abc, collections.abc, and numbers and expose names identical to the stdlib ones, so I could play around with it by just using abc2.isinstance, abc2.abc.ABCMeta, abc2.collections.abc.Mapping, etc. Or I could just toss some "from abc2 import"s around. But for any serious use beyond playing around, that would be quite confusing.
On 4 December 2013 18:19, Andrew Barnert <abarnert@yahoo.com> wrote:
It should also be based on the rich function inspect.signature API (introduced in http://www.python.org/dev/peps/pep-0362/), not on the older low level interfaces (the function signature API has been backported to Python 2.x as the funcsigs module on PyPI). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Thank you for taking time to answer. I trust you, but I've sometimes the feeling we're talking about different things. I didn't know PEP 3124 and PEP 0362, they are great ressources. Signature is particularly usefull, it's a pity it's not documented, I'll suggest something further. I'm ok with what you said (I talked effectively about signatures). The matter I see, is that attributes and methods can be set dynamically (that's why I talked about runtime), either I don't manage these cases, or I have to extend ABCMeta and override __instancecheck__ and __subclasscheck__. The second solution would make duplicate code unless I extract ABCMeta cache and registry operations in a dedicated object. This would be a lot of change and despite _abc_* attributes are protected, I'm afraid some library use them. I've made an example of failure to illustrate dynamic setting matter: https://gist.github.com/apieum/7800992 Maybe it's more effective for attributes which are often defined in __init__ uniquely. Whereas all of this is really interesting we are quite far from the initial target which was simply to avoid these cases:
I'll dig if I can find a better proposal considering all you suggestion. 2013/12/4 Andrew Barnert <abarnert@yahoo.com>
On 5 December 2013 16:48, Gregory Salvan <apieum@gmail.com> wrote:
It's covered in the inspect module docs: http://docs.python.org/3/library/inspect#introspecting-callables-with-the-si...
Use ABCMeta.__subclasshook__, that's what it is for. It's deliberate that ABCs don't check for instance attributes, and it's only the automatic caching behaviour (required for reasonable performance) that prevents the ABC machinery from recognising dynamic class updates by default. However, you *can* make it recognise updates by invalidating the caches (by doing an explicit registration on any ABC), and I'd be open to an "abc.invalidate_caches()" function as a more obvious way to force cache invalidation. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
It's covered in the inspect module docs:
Oups, I've missed it was introduced in v3.3
I'd be open to an "abc.invalidate_caches()" function as a
more obvious way to force cache invalidation.
Does this implementation make sense ? https://gist.github.com/apieum/7805431
On 12/02/2013 04:55 PM, Gregory Salvan wrote:
Just a question: I wonder about actual usefulness. For a strict safe check you'd also want to control method param types and names, not only number. Indeed, param types are out of question here, but what about param names? Since you implement that in C, is it at all possible to get method param names? (Well, in fact, maybe even Python's self-examination power allows that, does it?) Denis
Sounds like a cute thing to post on PyPI and see how popular it becomes. You have until the 3.5 release cycle starts to gather votes and improvements. There are lots of edge cases I could see going wrong, so I'm kind of skeptical of having this in the stdlib without strong caveats about its usefulness. Maybe it could go in the inspect module, which is traditionally full of heuristics. On Mon, Dec 2, 2013 at 7:55 AM, Gregory Salvan <apieum@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido)
Hi! On Mon, Dec 02, 2013 at 04:55:22PM +0100, Gregory Salvan <apieum@gmail.com> wrote:
Isn't the requirement too strong? When I need a file-like object I seldom need more than .read() and .write() methods, and even of those I seldom need both at once -- I usually need one or the other. Testing for strict conformance is not duck typing IMO -- it's interface. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Tue, Dec 3, 2013 at 3:11 AM, Oleg Broytman <phd@phdru.name> wrote:
But you could have a "ReadableFile" that has a .read() method and whatever else you need, thus using that class as a sort of interface. With a little careful scripting and introspection, you might even be able to craft that by source code analysis, which would be extremely cool (eg you run this script over your source code and it notes that elements of this list are indexed with these five keywords, so it constructs a class with those five so you can check what goes into the list). Whether it's actually _useful_ or not remains to be seen, but it would certainly be cool. ChrisA
Ok for pypi. (I've never tried to make python C extension, I feel it's going to be epic :) ) @Oleg you can override default behaviour with __ducktypecheck__ it's not as strict as an interface as var names can be differents and can have defaults. then instead of declaring an interface with "read" and "write", you just create a base class with these methods. What solution do you suggest ? I've made a function that returns public members only, maybe this method can be exposed in the api too. 2013/12/2 Oleg Broytman <phd@phdru.name>
On Tue, Dec 3, 2013 at 3:33 AM, Gregory Salvan <apieum@gmail.com> wrote:
Ok for pypi. (I've never tried to make python C extension, I feel it's going to be epic :)
Is there any reason this can't be implemented in pure Python? It'd be a lot easier to figure things out that way. Maybe a C implementation could come later, but for a first cut, this ought to be possible in Python. ChrisA
On Mon, Dec 02, 2013 at 05:33:10PM +0100, Gregory Salvan <apieum@gmail.com> wrote:
I don't suggest any solution as I don't have a problem to solve with interfaces. You are going to implement another interface checker in addition to existing ones? https://pypi.python.org/pypi?%3Aaction=search&term=interface&submit=search No objections from me. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Mon, Dec 02, 2013 at 08:04:19PM +0200, ?????????? <elazarg@gmail.com> wrote:
I'm sure the terminology in Python is quite stable. There are interfaces (there are many implementations, there is one rejected PEP and one deferred PEP) and there is duck typing. Formalizing duck typing is IMO constructing an interface. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
02.12.13 17:55, Gregory Salvan написав(ла):
class FooReader: def read(self, n=None): ... class BarReader: def read(self, amount=-1): ... class BazReader: def read(self, size=-1, chars=-1, firstline=False): ... Instances of all three classes can be used where needed an object with the read() method which are called with one positional argument or without arguments. Is it possible to use isducktype() for such case?
@Serhiy Not really (only with FooReader, BarReader) as it checks if functions co_argcount and co_kwonlyargcount are equals. In this case it requires BazReader.__ducktypecheck__ to be implemented. I had a doubt when doing this, I agree with you isducktype(BazReader, FooReader) can return True (whereas isducktype(FooReader, BazReader) should stay false) This can easily be changed. @Oleg: The fact there is a lot of interface implementations means certainly it's something needed. Each implementation I've seen add burden on objects whereas the only thing personnally I need, is to be sure I'll be able to call some methods off injected objects. @spir: Names can easily be checked, but I can't see why doing this, it add constraints to developpers without adding safety. @Andrew +1 Maybe inspect is more appropriate as it provides ArgSpec and getmembers ? My code is crappy, I've just do it like that to train me in C then tought it might be interesting to get feedback. 2013/12/2 Serhiy Storchaka <storchaka@gmail.com>
On Dec 2, 2013, at 17:21, Gregory Salvan <apieum@gmail.com> wrote:
@Andrew +1 Maybe inspect is more appropriate as it provides ArgSpec and getmembers ?
Sure, using inspect for the implementation _also_ makes sense. But my point is more about the interface to your new thing. Your thing checks most of the same stuff that ABCs check, but maybe not all of it, and also adds checking for compatible argspecs. (I believe that's the only new feature it adds; am I right?) Meanwhile, it uses a new isducktype function instead of working with normal issubclass/isinstance, and a completely different way of registering explicit compliance, and so on. Instead, why not just extend ABCMeta with a new subclass that just adds the check for compatible argspecs (maybe using inspect, as you say) and takes advantage of all the other stuff ABC already does? Besides being less code, which isn't that big of a deal, it's less of a radical conceptual change, which is a bigger deal, and it means that all kinds of code that relies on isinstance today can work with no changes.
On 3 December 2013 04:42, Andrew Barnert <abarnert@yahoo.com> wrote:
Shouldn't this be tied to ABCs rather than redesigning and reimplementing most of ABC to add a little bit on top?
This seems like an apropos place for this link: http://docs.python.org/3/library/abc#abc.ABCMeta.__subclasshook__ That's the existing "formalised ducktyping" hook, that already allows things like the following:
The advantage ABCs have over ducktyping alone is that subclassing and explicit registration allow ambiguities like the one between the Sequence and Mapping interfaces to be resolved (you can't always just ducktype those, as they have the same methods - they differ only in how the __*item__ methods handle slice objects). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Sorry Andrew, I'm not sure to understand all your point. I don't believe it reimplements things of ABC, or I don't see what. I agree ABC solution is the cleanest, whereas I would be able to check functions, stay on a runtime checking and being free to define or not a kind of protocol. Before reading your comments on ABC, I've released the python version of "isducktype": https://github.com/apieum/ducktype I've modified some behaviours within your comments and made it more stable. It would be interesting to have the version wich use ABCMeta, so I'm going to implement it. I've thought to name it SignedMeta, or ProtocolMeta, any opinion ? 2013/12/3 Nick Coghlan <ncoghlan@gmail.com>
On Dec 3, 2013, at 21:22, Gregory Salvan <apieum@gmail.com> wrote:
Sorry Andrew, I'm not sure to understand all your point. I don't believe it reimplements things of ABC, or I don't see what.
Almost everything you did reimplements features of ABCs. The builtin isinstance and issubclass functions can check whether you implement the methods of a type, and you can override that by explicitly declaring that you implement the type. That's the core feature set of your proposal, and of ABCs; you've just implemented the same thing with a different API, plus and minus a few minor details. If you haven't read PEP 3119, you really should. (It's also worth reading rejected predecessors like 245/246 and successors like 3124, and of course the pre-existing third party stuff like PyProtocols and the Interface types in Twisted and Zope.)
I agree ABC solution is the cleanest, whereas I would be able to check functions, stay on a runtime checking and being free to define or not a kind of protocol.
I don't understand what you mean here. ABCs can check methods. They do so at runtime. You're free to define or not define any protocol you want. What you have to add is checking the argspecs of the methods rather than just their names. That's a great idea, but it's a much better idea as an addition to what's already there than as a complete parallel API to very similar functionality. And if this were added to the stdlib, I'm pretty sure it would be a modification/extension of the abc module, not a competing and similar but not identical module.
First, it sounds like you're focusing on the wrong part of the name--ABCMeta is a metaclass for implementing ABCs; leaving out the ABC part sounds like you're designing a metaclass for something completely unrelated. Less seriously, ProtocolMeta sounds like it's going to add more PyProtocols-style functionality (in particular, adapters from one protocol to another), but there's only so many synonyms available and they're all taken, so I don't think that's a major problem. As for SignedMeta, I don't even know what that's meant to imply. Is it to do with method signatures? If so, I think "signature" is a better word; you don't really think about anything being "signed" in a method signature; that's dragging in associations from a different meaning of the word. But I don't have any better suggestions. Protocol, interface, prototype, type, abstract type, abstract base... As I said, all of the names have already been used up, repeatedly, and none of them have any particular connotations that would explain how your module, metaclass, function, etc. differ from abc. In fact, I'd probably just come up with a stupid package name like abc2 and stick in modules named abc, collections.abc, and numbers and expose names identical to the stdlib ones, so I could play around with it by just using abc2.isinstance, abc2.abc.ABCMeta, abc2.collections.abc.Mapping, etc. Or I could just toss some "from abc2 import"s around. But for any serious use beyond playing around, that would be quite confusing.
On 4 December 2013 18:19, Andrew Barnert <abarnert@yahoo.com> wrote:
It should also be based on the rich function inspect.signature API (introduced in http://www.python.org/dev/peps/pep-0362/), not on the older low level interfaces (the function signature API has been backported to Python 2.x as the funcsigs module on PyPI). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Thank you for taking time to answer. I trust you, but I've sometimes the feeling we're talking about different things. I didn't know PEP 3124 and PEP 0362, they are great ressources. Signature is particularly usefull, it's a pity it's not documented, I'll suggest something further. I'm ok with what you said (I talked effectively about signatures). The matter I see, is that attributes and methods can be set dynamically (that's why I talked about runtime), either I don't manage these cases, or I have to extend ABCMeta and override __instancecheck__ and __subclasscheck__. The second solution would make duplicate code unless I extract ABCMeta cache and registry operations in a dedicated object. This would be a lot of change and despite _abc_* attributes are protected, I'm afraid some library use them. I've made an example of failure to illustrate dynamic setting matter: https://gist.github.com/apieum/7800992 Maybe it's more effective for attributes which are often defined in __init__ uniquely. Whereas all of this is really interesting we are quite far from the initial target which was simply to avoid these cases:
I'll dig if I can find a better proposal considering all you suggestion. 2013/12/4 Andrew Barnert <abarnert@yahoo.com>
On 5 December 2013 16:48, Gregory Salvan <apieum@gmail.com> wrote:
It's covered in the inspect module docs: http://docs.python.org/3/library/inspect#introspecting-callables-with-the-si...
Use ABCMeta.__subclasshook__, that's what it is for. It's deliberate that ABCs don't check for instance attributes, and it's only the automatic caching behaviour (required for reasonable performance) that prevents the ABC machinery from recognising dynamic class updates by default. However, you *can* make it recognise updates by invalidating the caches (by doing an explicit registration on any ABC), and I'd be open to an "abc.invalidate_caches()" function as a more obvious way to force cache invalidation. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
It's covered in the inspect module docs:
Oups, I've missed it was introduced in v3.3
I'd be open to an "abc.invalidate_caches()" function as a
more obvious way to force cache invalidation.
Does this implementation make sense ? https://gist.github.com/apieum/7805431
On 12/02/2013 04:55 PM, Gregory Salvan wrote:
Just a question: I wonder about actual usefulness. For a strict safe check you'd also want to control method param types and names, not only number. Indeed, param types are out of question here, but what about param names? Since you implement that in C, is it at all possible to get method param names? (Well, in fact, maybe even Python's self-examination power allows that, does it?) Denis
participants (10)
-
Andrew Barnert -
Chris Angelico -
Gregory Salvan -
Guido van Rossum -
Jan Kaliszewski -
Nick Coghlan -
Oleg Broytman -
Serhiy Storchaka -
spir -
אלעזר