Just FYI there is a closely related issue on b.p.o. https://bugs.python.org/issue25988. FWIW I am in favor of the idea, but some people are against it (see e.g. the issue). -- Ivan On Fri, 27 Sep 2019 at 17:12, Steven D'Aprano wrote:

I keep finding myself needing to test for objects that support subscripting. This is one case where EAFP is *not* actually easier:

try: obj[0] except TypeError: subscriptable = False except (IndexError, KeyError): subscriptable = True else: subscriptable = True if subscriptable: ...

But I don't like manually testing for it like this:

if getattr(obj, '__getitem__', None) is not None: ...

because it is wrong. (Its wrong because an object with __getitem__ defined as an instance attribute isn't subscriptable; it has to be in the class, or a superclass.)

But doing it correctly is too painful:

if any(getattr(T, '__getitem__', None) is not None for T in type(obj).mro())

and besides I've probably still got it wrong in some subtle way. What I'd really like to do is use the collections.abc module to do the check:

if isinstance(obj, collections.abc.Subscriptable): ...

in the same way we can check for Sized, Hashable etc.

Alternatively, if we had a getclassattr that skipped the instance attributes, I could say:

if getclassattr(obj, '__getitem__', None) is not None: ...

(1) Am I doing it wrong? Perhaps I've missed some already existing solution to this.

(2) If not, is there any reason why we shouldn't add Subscriptable to the collection.abc module? I think I have the implementation:

class Subscriptable(metaclass=ABCMeta): __slots__ = () @abstractmethod def __getitem__(self, idx): return None @classmethod def __subclasshook__(cls, C): if cls is Subscriptable: return _check_methods(C, "__getitem__") return NotImplemented

Comments, questions, flames?

-- Steven _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/JCQJP5... Code of Conduct: http://python.org/psf/codeofconduct/

On Sep 27, 2019, at 09:05, Steven D'Aprano wrote:

What I'd really like to do is use the collections.abc module to do the check:

if isinstance(obj, collections.abc.Subscriptable): ...

What about isinstance(obj, (Sequence, Mapping))? That isn’t quite the same thing, since you can have types that are subscriptable, but its subscripts don’t mean either index or key. _GenericAlias is probably not the best example here, but it is an example. But it is close to the same thing. I can’t think of too many cases where you want to work with sequences and mappings and generic static types the same way. Plus, it seems like it’s actually a more direct LBYL translation of what you’re EAFPing with your except (IndexError, KeyError) test. I think the only real issue is that it’s no too uncommon to create a type that acts like a sequence but doesn’t register with Sequence. Maybe that’s enough to make it unusable for some important uses? Your Indexable class could be implicit (like Sized, Iterable, etc.), which would solve that problem if it’s a problem (at the cost of possibly unintentionally including things like _GenericAlias, but that may not be much cost—and may not even be unintentional?). Or I may be missing something that makes (Sequence, Mapping) inappropriate, of course.

On Sep 27, 2019, at 10:15, MRAB wrote:

...

On 2019-09-27 17:05, Steven D'Aprano wrote:

But doing it correctly is too painful: if any(getattr(T, '__getitem__', None) is not None for T in type(obj).mro()) [snip]

Is there are reason why you're using 'getattr' instead of 'hasattr'?

There is a difference here: if some class defines __getitem__ = None, blocks it from looking like a subscriptable type. That idiom is used for other things like __hash__, even if it’s not used 100% consistently for all protocols.

On Fri, Sep 27, 2019 at 10:11:15AM -0700, Andrew Barnert wrote:

On Sep 27, 2019, at 09:05, Steven D'Aprano wrote:

...

What I'd really like to do is use the collections.abc module to do the check:

if isinstance(obj, collections.abc.Subscriptable): ...

What about isinstance(obj, (Sequence, Mapping))?

No, they require more than just being subscriptable. py> class Test(object): ... def __getitem__(self, idx): ... return idx**2 ... py> squares = Test() py> squares[5] 25 py> from collections.abc import Sequence, Mapping py> isinstance(squares, (Sequence, Mapping)) False One of the tests I want is for something which is subscriptable but not sized, like squares above. To be a (virtual) subclass of Sequence, the object has to provide __getitem__ and __len__, and to be a Mapping it also has to provide __iter__. Neither __len__ nor __iter__ are necessary for my purposes, and in fact __len__ may be undesirable.

That isn’t quite the same thing, since you can have types that are subscriptable, but its subscripts don’t mean either index or key.

Its not even close to the same thing :-(

_GenericAlias is probably not the best example here, but it is an example.

Where is that from?

Plus, it seems like it’s actually a more direct LBYL translation of what you’re EAFPing with your except (IndexError, KeyError) test.

Sorry, that doesn't fly. There's nothing in my except test which requires the existence of __len__. All I need is an object that supports subscripting, but the current state of the ABCs requires that I either roll my own test which will probably be wrong, or test for methods that I don't need or want. -- Steven

On Sun, 29 Sep 2019 at 08:28, Kyle Stanley wrote:

Raymond also brought up a strong point to consider in https://bugs.python.org/issue25988:

...

The OP has a sense that Mapping and Sequence are "too heavy" but I think the reality that useful classes almost never use __getitem__ in isolation; rather, it is part of a small constellation of methods that are typically used together.

That's the point that I would make as well. What can you do with an object that is only known to be Subscriptable? With a Mapping I know that `keys` or `__iter__` give me the subscripts that can be used with it. With a Sequence `__len__` tells me the valid subscripts. Either of those makes it possible for me to build a sensible algorithm using subscripts. Without either of those what can I do in practice in a situation like this: def do_subscriptable_things(obj): if isinstance(obj, Subscriptable): # Now what? -- Oscar

On Sun, Sep 29, 2019 at 5:04 AM Ricky Teachey wrote:

That's the point that I would make as well. What can you do with an

...

object that is only known to be Subscriptable?

...

def do_subscriptable_things(obj): if isinstance(obj, Subscriptable): # Now what?

I'm going to echo this one -- the OP has shown that it's awkward to know whether an arbitrary object is "subscriptable" -- but not why you'd want to do that check at all. In fact, you do not use "EAFTP" to do a type check -- the whole point is that you use it like you expect ot use it, and if you get an error, you know it can't be used that way: So the OP's code: try: obj[0] except TypeError: subscriptable = False except (IndexError, KeyError): subscriptable = True else: subscriptable = True if subscriptable: ... Is exactly NOT EAFTP -- it's LBYL, but using exception catching to do the type check. And after this code is run, now what? you don't know anything about HOW you can subscript that object -- can you use an index, can you use a key, can you do something really wierd and arbitrary: Maybe if you want to use/abuse it as an alternative function calling

syntax-- square brackets rather than parentheses.

Like the example given a bit earlier: class Test(object): def __getitem__(self, idx): return idx**2 py> squares = Test() py> squares[5] 25 So if you think you *may* have a "callable_with_an_integer" -- the thing to do is: try: result = squares[5] except TypeError: print("Opps, couldn't do that") Though what you'd probably really want to do is have your oddball class raise appropriate exceptions is the wrong kind of "index" was passed in. If you do want to do this, maybe a "callable_with_brackets" ABC ??? (but please don't -- there is already a way to make objects callable...) Now that I think about it a bit more, that example *could* make sense, but only if what you are trying to do is make an indexable lazy-evaluated infinite sequence of all the squares -- in which case, you'd want to put a bit more in there. And now that I think about it, there doesn't appear to be an ABC for something that can be indexed and iterated, but does not have a length. However, we need to think more about what ABCs are *for* anyway -- given Python's "magic method" system and Duck Typing, you can essentially create types that have any arbitrary combination of functionality -- do we need an ABC for every one? Obviously not. When I first learned about ABCs, I thought some about what the point was -- they are NOT the same as, say, ABCs in C++ -- Python doesn't need those. I came to the conclusion that the point of ABCs was to allow users to write code that expects standard built-ins that will also work with custom types that match that interface. But it's only useful for standard Base Classes -- making a custom ABC that matches the spec of a particular use case isn't very helpful -- unless various third parties are going to be using that spec, there's not point in an ABC. Modifying my point -- ABCs are useful for built ins or large projects with a wide user base that may have third parties writing extensions, plug ins, etc). However, if a spec gets complicated, the ABC system breaks down anyway -- go look at what numpy is trying to do with "duck arrays" -- ABCs are not very helpful there. In short: in python, __getitem__ can be [ab]used for virtually ANYTHING -- so if you have an object that isn't a mapping (KeyError), or a Sequence (IndexError), they you would need to know something more about what it is anyway in order to use it -- I can't see how simply knowing that you can toss some unknown type(s) of objects into the square brackets is useful. So back to the OP: What do you want to be able to do here? What types do you want to be able to support, and how do you want to use them? -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On Sun, Sep 29, 2019 at 09:05:19AM -0700, Christopher Barker wrote:

And after this code is run, now what? you don't know anything about HOW you can subscript that object -- can you use an index, can you use a key, can you do something really wierd and arbitrary:

That applies to many interface checks. Both the built-in callable() and the Callable ABC don't tell you what arguments the object accepts, or what it does; the Sequence ABC check doesn't tell you what valid indexes are accepted. People may assume that valid indexes go from 0 through len-1 but we can make sequences that work with 1-based sequences, or like Fortran, with any bounds at all. Including virtual "sequences" that accept indexes from -infinity to infinity, with no length. "Consenting adults" applies. The ABC checks only test for the presence of an interface, they don't necessarily make guarantees about what the interface will do.

So if you think you *may* have a "callable_with_an_integer" -- the thing to do is:

try: result = squares[5] except TypeError: print("Opps, couldn't do that")

I know that's meant in good faith, but please be careful of telling me what I should do on the basis of a wild guess of what my needs are. I've been programming in Python since 1.5 was the latest and greatest, and I think I'm capable of judging when I want to use a try...except EAFP check rather than a LBYL check for an interface. And even if I personally am not, this is not about *me*. The community as a whole is, which is why Python supports ABCs and isinstance. If you want to argue against that, that ship has not only sailed but it's already docked at the destination and the passengers disembarked :-)

Now that I think about it a bit more, that example *could* make sense, but only if what you are trying to do is make an indexable lazy-evaluated infinite sequence of all the squares --

Well spotted :-)

in which case, you'd want to put a bit more in there.

I'm listening.

And now that I think about it, there doesn't appear to be an ABC for something that can be indexed and iterated, but does not have a length.

Exactly! Don't forget that you don't need __iter__ or length to be iterable: py> class Example: ... def __getitem__(self, idx): ... if idx in range(0, 4): return "spam"*idx ... raise IndexError ... py> list(Example()) ['', 'spam', 'spamspam', 'spamspamspam']

However, we need to think more about what ABCs are *for* anyway -- given Python's "magic method" system and Duck Typing, you can essentially create types that have any arbitrary combination of functionality -- do we need an ABC for every one? Obviously not.

I don't think it is so obvious. Clearly we can't have an ABC for every imaginable combination of magic methods. There would be hundreds. What would we name them? We should (and do!) have ABCs for the most common combinations, like Sequence, MutableMapping, Container etc. And we should (so I am arguing) have ABCs for the underlying building blocks, the individual magic methods themselves. Developers can then combine those building blocks to make their own application-specific combinations. But without those composable building blocks, we're reduced to testing for dunders by hand, which is problematic because: - people get it wrong, using ``hasattr(instance, dunder)`` - ``getattr(type(instance), dunder, None) is not None`` is apparently still wrong, since that's not what collections.abc does, and presumably it has a good reason for not doing so. - being part of the implementation, rather than public interface, in principle dunders may change in the future. Obvious well-known dunders like __len__ will probably never go away. But the obvious and well-known dunder __cmp__ went away, and maybe one day the Sized protocol will change too. If you want to future-proof your code, you should test for the Sized interface rather than directly testing for the __len__ magic method. -- Steven

On 2019-09-29 21:15, Steven D'Aprano wrote:

Clearly we can't have an ABC for every imaginable combination of magic methods. There would be hundreds. What would we name them?

We should (and do!) have ABCs for the most common combinations, like Sequence, MutableMapping, Container etc.

And we should (so I am arguing) have ABCs for the underlying building blocks, the individual magic methods themselves. Developers can then combine those building blocks to make their own application-specific combinations.

But without those composable building blocks, we're reduced to testing for dunders by hand, which is problematic because:

- people get it wrong, using ``hasattr(instance, dunder)``

- ``getattr(type(instance), dunder, None) is not None`` is apparently still wrong, since that's not what collections.abc does, and presumably it has a good reason for not doing so.

- being part of the implementation, rather than public interface, in principle dunders may change in the future.

Obvious well-known dunders like __len__ will probably never go away. But the obvious and well-known dunder __cmp__ went away, and maybe one day the Sized protocol will change too. If you want to future-proof your code, you should test for the Sized interface rather than directly testing for the __len__ magic method.

I agree with this reasoning. I tend to be more of a "special cases aren't special enough to break the rules" kind of guy than a "practicality beats purity kind of guy". From my perspective, it doesn't really matter what you would or wouldn't use this ABC for. The ABC module is desirable at a conceptual level for providing just the type of implementation-agnostic facility you describe for querying the potential interface capabilities of objects. Getting items is a basic potential interface capability of objects and that alone is enough is enough justification for adding an ABC for it. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

On Mon, 30 Sep 2019 at 10:02, Stephen J. Turnbull wrote:

Steven D'Aprano writes:

...

On Sun, Sep 29, 2019 at 11:27:32AM +0100, Oscar Benjamin wrote:

...

That's the point that I would make as well. What can you do with an object that is only known to be Subscriptable?

I can subscript it. What did you expect the answer to be?

Technical questions: does "Subscriptable" mean non-negative ints only, or does it include the negative "count from the end" protocol? How about slices?

I think we've established that what Steven wants is nothing more than a "robust" version of hasattr(obj, "__getitem__"). It's (in my view) sad that the simple hasattr test is no longer sufficient, and in particular that if you want robustness, Python has changed to the point where a pseudo subclass check is the "right" way to check for an object that has certain properties. But I guess this is the case, and therefore the omission of a Subscriptable ABC is something that may need to be addressed. Maybe rather than proliferating ABCs like this, exposing the internal function in the ABC that does the "robust" version of the hasattr test would be more flexible? I guess that depends on whether you find the "subclass of an ABC" approach acceptable. To me, it feels too much like "object oriented everywhere" languages like Java, which was always an issue I had with ABCs, but as Steven said, that ship has probably sailed by now[1]. Paul [1] Much like typing, where it's getting harder and harder if you work on larger projects to view it as "optional" when the project standards mandate it...

...

That applies to many interface checks. Both the built-in callable() and the Callable ABC don't tell you what arguments the object accepts, or what it does; the Sequence ABC check doesn't tell you what valid indexes are accepted.

Sure — and Callable is probably the closest ABC to this proposal there is. But while I’m having trouble putting my finger on it, Callable seems to have a special place in the language that makes it important. While __getitem__ could do just about anything, I’d bet most folks would consider to be bad form to use it for something that did not “feel” like indexing. People may assume that valid indexes go from 0 through

len-1 but we can make sequences that work with 1-based sequences, or like Fortran, with any bounds at all.

Sure, but they should all raise IndexError if you violate the rules. Including virtual "sequences" that accept indexes from -infinity to

infinity, with no length.

"Consenting adults" applies. The ABC checks only test for the presence of an interface, they don't necessarily make guarantees about what the interface will do.

Sure, but again, what’s the point here? What’s the use case for knowing that the object at hand will allow you to pass something into the square brackets, but nothing else?

So if you think you *may* have a "callable_with_an_integer" -- the thing to

...

do is:

try: result = squares[5] except TypeError: print("Opps, couldn't do that")

I know that's meant in good faith, but please be careful of telling me what I should do on the basis of a wild guess of what my needs are. I've been programming in Python since 1.5 was the latest and greatest, and I think I'm capable of judging when I want to use a try...except EAFP check rather than a LBYL check for an interface.

I was using “you” rhetorically— but more importantly, I was suggesting that that was the how to do EAFTP, not that you should do EAFTP. The OP has posted a more convoluted form. why Python supports ABCs and isinstance. If you

want to argue against that, that ship has not only sailed but it's already docked at the destination and the passengers disembarked :-)

Interestingly, use of ABCs seems to be really growing with static type checking — relatively new.

Now that I think about it a bit more, that example *could* make sense, but

...

only if what you are trying to do is make an indexable lazy-evaluated infinite sequence of all the squares --

Well spotted :-)

So then you want an “IndexableIterable” ABC or some such. That might be a useful ABC. Why do I think it would be more useful? Because it would specify more than just “you can put any old thing into the square brackets”. you can essentially create

...

types that have any arbitrary combination of functionality -- do we need an ABC for every one? Obviously not.

I don't think it is so obvious.

Clearly we can't have an ABC for every imaginable combination of magic methods. There would be hundreds. What would we name them?

That’s why I said it was obvious :-)

And we should (so I am arguing) have ABCs for the underlying building blocks, the individual magic methods themselves. Developers can then combine those building blocks to make their own application-specific combinations.

But this is s very different proposal! If I have it right, you are now suggesting that there be an ABC for every magic method (there are a LOT). Why? So we can use isinstance() to check for a particular protocol, rather than try:except or hasattr() But this seems to be really stretching what Python ABCs are about. OK, that ship has sailed, and no, I’m not particularly happy about it. But you can make your own ABC with the particular combination of attributes you want — wouldn’t that make more sense than asking your users to call isinstance with 10 ABCs? Which brings me back to the original proposal: 1) why not make your own ABC? And 2) can someone provide even a single use case? We’ve seen one example of a class that implemented __getitem__ in an unconventional way, but not an example of how having an ABC for it would be actually useful. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On Mon, Sep 30, 2019 at 06:00:44PM +0900, Stephen J. Turnbull wrote:

Technical questions: does "Subscriptable" mean non-negative ints only, or does it include the negative "count from the end" protocol? How about slices?

It means the class defines a __getitem__ method. Like __call__, the semantics of that method, and the range of acceptable arguments, is out of scope of the ABC. I don't believe that Python has any way to check the semantics of a method call, except to call it and see what it does. -- Steve (the other one)

On Tue, Oct 1, 2019 at 8:48 AM Ben Rudiak-Gould mailto:benrudiak@gmail.com> wrote:

Is it? Subtyping in type theory satisfies some axioms, one of which is transitivity. The addition of the ABCs broke transitivity:

>>> issubclass(list, object) True >>> issubclass(object, collections.abc.Hashable) True >>> issubclass(list, collections.abc.Hashable) False

This isn't really the fault of ABCs, it's a consequence of the fact that subclasses can nobble certain methods by setting them to None. The non-transitivity was already there, it's just that issubclass() doesn't always detect it. -- Greg

On Oct 1, 2019, at 06:23, Richard Musil wrote:

Where *virtual* is a hyperlink to an ABC definition.

`issubclass` will be fine without virtual and I am not sure why virtual has been added to it

Virtual subclassing is key to the way ABCs work. It would be nice if it were explained better in the docs, but I suppose usually people don’t need to know the details unless they’re actually getting involved in something like designing new ABCs, so… The general feature is that any class can define a subclass-testing dunder method to tell issubclass what to do, and likewise for isinstance. That’s what virtual subclassing means: your class’s subclass hook can say that something is a subclass even though it didn’t inherit from you. This is what allows ABCs like Iterable to simulate structural subtyping. In Java, you can implement all the methods you want, but you won’t be usable as an iterable unless you also inherit from the IIterable interface. In Go, if you implement the right methods, you are automatically usable as an Iterable without inheriting anything. ObjC allows you to define both kinds of relationships, because sometimes one is useful, sometimes the other. Python gives you a general framework that lets you define both kinds of relationships—and more; e.g., you can handle relationships that aren’t quite structurally (Go-style) checkable, like Sequence, and you can fudge around legacy issues rather than having to get everything perfect before you can use new features. And it does it without needing any special language-level magic like Go, by just treating issubclass itself (and likewise isinstance) as a protocol, with a dunder method, like __getattr__ or anything else. The ABC metaclass gives you some helpers to make it easier to implement virtual subclassing correctly and consistently, like a single simpler method you can override to handle both hooks at once, a registry to allow classes to explicitly claim they match your ABC, and a few other things. The collections.abc module adds an additional helper on top of that to help its ABCs test for methods properly (because it’s not as simple as it should be). But the real magic is making issubclass a protocol that’s overridable the same as everything else in Python.

On Tue, Oct 1, 2019 at 6:48 PM Andrew Barnert wrote:

Virtual subclassing is key to the way ABCs work. It would be nice if it were explained better in the docs, but I suppose usually people don’t need to know the details unless they’re actually getting involved in something like designing new ABCs, so…

Thanks for the concise explanation. I guess my objection was not about the ABC (or how it works), but about the single fact that the function `issubclass` gives the results which are counter intuitive. Yes, we know, that those three calls: >>> issubclass(list, object) True >>> collections.ishashable(object) True >>> collections.ishashable(list): False are correct, i.e. there is nothing wrong with each one individually. It is just that all together are contradicting. And it is not because of the objects (how they are), but because what the function does, and in particular that it does two different things (but pretends it is one). If `issubclass` tested the inheritance, and there was a function `collections.abc.islikeabc` doing the other thing, I would not say a word. The problem, as I perceive it, is not in a fact that a class can mangle its dunder methods in a way it becomes "contradicting" itself, as in, for example, deriving from object and then disabling __hash__, or deriving from list and removing __getitem__, but in the fact that someone decided to merge two different and basically independent functionalities into one function. And these functionalities are different, because they answer two different questions. First is about the class origin (heritage), the other is about the class behavior (interface), where neither one implicate necessarily the other, but I digress. For what concerns "issubscriptable" my only comment is that so far it does not seem there is a consensus about what exactly "subscriptable" means (or should mean) and the regular user (like myself) could probably be even more confused (like I am about `issubclass` right now). Richard

On Wed, Oct 2, 2019 at 5:18 AM Ben Rudiak-Gould wrote:

On Tue, Oct 1, 2019 at 9:26 AM Andrew Barnert wrote:

...

On Sep 30, 2019, at 23:46, Ben Rudiak-Gould wrote:

...

ABC membership is a subtype relationship in some sense, and ordinary Python subclassing is a subtype relationship in some sense, but they aren't quite the same sense,

But in this case, they actually match. Hashable is correctly checking for structural subtyping, and the problem is that list isn’t actually a proper subtype of object, not that object isn’t a proper subtype of Hashable.

If you take the perspective that the ABC notion of subtyping is the correct one, then list isn't a proper subtype of object. I wasn't taking the perspective that either one is the correct one. I think they're probably both okay in isolation.

If you want to get REALLY technical, very few types - if any - are actually proper subtypes. A strict reading of the Liskov Substitution Principle would imply that the repr of every object should remain identical to object.__repr__ of that object. That's completely useless, of course, but the question is: if a subclass is allowed to make changes to its parent's behaviour, where do you draw the line? For instance, it is extremely common to expect that x == x for any object x, but float is a subtype of object, and not every float maintains that property. The base object type compares by identity, so object() != object(). Is it violating LSP if two non-identical objects compare equal? The base object is immutable. Subtypes that are also immutable will follow the principle that if a == b now, then a == b forever. Is it violating LSP to have mutable objects that can compare equal now but unequal later? The hashability issue is a logical consequence of accepting that the above violations are reasonable and practically useful. Since a==b must imply that hash(a)==hash(b), it logically follows that either the hash of every list is the same (thus destroying the value of hashing at all), or the hash of a list could change (thus destroying the very meaning of object hashes). I'm not sure what would break if every list object had a hash of 0x142857, but certainly it wouldn't be of much practical use, and would be a performance nightmare (you use a list as a dict key and suddenly every lookup is a linear search). Yes, it's not a "proper subtype", but proper subtypes are impractically restrictive. ChrisA

Chris Angelico wrote:

The hashability issue is a logical consequence of accepting that the above violations are reasonable and practically useful.

A more principled way to handle this wouild be for object not to be hashable, and have another base type for hashable objects. Hashable would then be a subtype of object, not the other way around. -- Greg

On Oct 1, 2019, at 11:53, Richard Musil wrote:

...

On Tue, Oct 1, 2019 at 6:48 PM Andrew Barnert wrote:

...

Virtual subclassing is key to the way ABCs work. It would be nice if it were explained better in the docs, but I suppose usually people don’t need to know the details unless they’re actually getting involved in something like designing new ABCs, so…

Thanks for the concise explanation. I guess my objection was not about the ABC (or how it works), but about the single fact that the function `issubclass` gives the results which are counter intuitive.

But it only gives counterintuitive results when you do something counterintuitive. That’s how all consenting-adults features work:

Yes, we know, that those three calls:

>>> issubclass(list, object) True >>> collections.ishashable(object) True >>> collections.ishashable(list): False

are correct, i.e. there is nothing wrong with each one individually. It is just that all together are contradicting. And it is not because of the objects (how they are), but because what the function does, and in particular that it does two different things (but pretends it is one).

Not really. If you want to be a purist, Python doesn’t do subtyping—or, at best, subtyping is defined as “whatever the subclass hooks say”, which is meaningless because it’s completely unrestricted. By the same argument, Python doesn’t do attributes, or at best attribution is defined as “whatever the getattribute/etc. hooks say”, which is meaningless because it’s completely unrestricted. But in practice, Python objects almost always act pretty much like things that have attributes. And in practice, Python objects almost always act pretty much like things that have subtyping relationships. The fact that Python allows you to do otherwise, and that there are occasional practical reasons to do otherwise, doesn’t really change that. (At least not for people trying to read and write Python code; for someone trying to write an automated correctness prover, it radically breaks everything, and for someone trying to work on the design of ABCs, it makes less difference than that but still more than zero.)

If `issubclass` tested the inheritance, and there was a function `collections.abc.islikeabc` doing the other thing, I would not say a word.

Inheritance is nearly always useless to test for. And islikeabc would also be nearly always useless. The only reason either one is worth testing for is to approximate subtyping. You know that when isinstance(spam, Eggs) is true, you can use spam as an Eggs and it will generally work. That’s what you care about. You don’t care why it’s true, you just care that it’s true. And it’s generally (but not absolutely always, much less provably always) true if at least one of the following is true: * Eggs is a base class of type(spam) * Eggs is a structural-testing ABC where type(spam) meets the test * Eggs is an ABC where someone explicitly called Eggs.register on type(spam) * Eggs is a class with a complicated instance hook that does something very unusual but appropriate for this special case, and that unusual thing likes spam And that’s why there’s a single function that returns true if any of those is true. None of those things on its own is useful; all of them are useful as approximations of subtyping; all of them combined are in practice a more useful approximation of subtyping than any of them on its own.

For what concerns "issubscriptable" my only comment is that so far it does not seem there is a consensus about what exactly "subscriptable" means (or should mean) and the regular user (like myself) could probably be even more confused (like I am about `issubclass` right now).

Python already defines the term “subscription” for the syntactic form spam[eggs], and has done so consistently since 1.x. And it’s not a word that has other meanings in the Python community, or in programming in general. Given that, the most obvious meaning of “subscriptable” is “can be used as the left thing in a subscription”. And that’s what Steven is asking for here. The problem here is not that you’re confused about the meaning of subscriptable, but that it’s a concept that comes up rarely enough that you never even learned it. Which could be an argument that subscriptable isn’t useful. Or that we really need indexable and keyable as well/instead. Or even that Steven is wrong to think that he wants subscriptable, and what he really wants is indexable, so we don’t even have a use case for subscriptable. (I don’t know that these are very compelling arguments, but at least they’re conceivable ones.) But I can’t see an argument that something called Subscriptable could (or should) actually mean indexable rather than subscriptable.

On Wed, Oct 2, 2019 at 8:51 AM Greg Ewing wrote:

Chris Angelico wrote:

...

The question then would be: why is object() not hashable?

It's not hashable because it's supposed to be the ultimate base type for all other objects, and not every object is hashable.

It only seems odd if you're used to the idea that you get a bunch of default behaviours from object, including hashability. But if you want strict subtyping and also an ultimate base type, the base type has to include very little behaviour.

Should the default object be comparable? Or would there be a "Comparable" subtype? If object() can be compared for equality with another object(), then it'd be illogical to make it unhashable - there's no reason to deny hashing. If it can't be compared, why not? Why should the base type include THAT little behaviour? Python's base object has a lot of extremely useful functionality, including being able to generate a string representation, introspect its attributes, pickle and unpickle it, and more. All of these features are available to all objects unless they choose not to. Do we really need separate subclasses for Picklable and Hashable, forcing every class author subclass all of them? Or can we allow most objects to be indeed pickled and hashed, and then open files choose to be unpickleable and lists choose to be unhashable? ChrisA

On Oct 1, 2019, at 12:17, Ben Rudiak-Gould wrote:

...

On Tue, Oct 1, 2019 at 9:26 AM Andrew Barnert wrote:

...

On Sep 30, 2019, at 23:46, Ben Rudiak-Gould wrote: ABC membership is a subtype relationship in some sense, and ordinary Python subclassing is a subtype relationship in some sense, but they aren't quite the same sense,

But in this case, they actually match. Hashable is correctly checking for structural subtyping, and the problem is that list isn’t actually a proper subtype of object, not that object isn’t a proper subtype of Hashable.

If you take the perspective that the ABC notion of subtyping is the correct one, then list isn't a proper subtype of object.

I don’t think anyone takes that perspective. The correct notion of subtyping for Python is the one that exactly reproduces duck typing in an LBYL-checkable way, which I’ll bet is provably equivalent to the halting problem. However, the whole mess of things that go into isinstance and issubclass in Python (including third-party code) together define a pretty-good-in-practice approximation of that, when user with most real-life code. And a much better one than either inheritance or structural tests alone.

My problem is not that I can't justify the current behavior, it's that if it behaved differently, I could justify that behavior too. I feel like you're using CPython as an oracle of what ABCs should do,

Well, I’m using all Python implementations, together with all of the popular third-party code out there, as used by thousands of people every day, allowing us to work with types in a way that’s more pleasant to read than Java or Go or Objective C or Ruby, as an argument that Python’s design is in practice pretty good. I can make a good argument that inheritance alone, or structural subtyping alone, would be so terrible as to be nearly useless for Python. I can’t make an argument that no possible thing you could think of could be better than what Python currently does. (In fact, I can think of things I’d suggest be done differently if I had a time machine back. Then again, I could improve even more things with even less work by taking that time machine even further back and giving Guido the 2.3 type system back in 0.9, or *args and **kw.) But I don’t see why I have to make an argument for a years-old feature just because someone wants to add a minor extension that goes along with the way that feature has always been used. If you hate ABCs, you’re not going to use collections.abc.Subscriptable now matter how it’s bikeshedded.

What *should* it mean to inherit from an ABC? The module encourages you to use them as mixins, but maybe that isn't the intended meaning, just a side hack? Is the primary meaning to do the equivalent of registering the class with the predicate?

The intended meaning clearly is a mix of multiple things: some of them do structural subtyping, some don’t; some rely heavily on registration; some act as useful mixins, some don’t; etc. Theoretically, abc is a bit of a mess, and collections.abc, numbers, and io are a horrible pile of unusable crap. But in practice, they actually work pretty nicely in a wide range of code that people read every day.

On Tue, 1 Oct 2019 at 11:24, Steven D'Aprano wrote:

On Mon, Sep 30, 2019 at 06:00:44PM +0900, Stephen J. Turnbull wrote:

...

Technical questions: does "Subscriptable" mean non-negative ints only, or does it include the negative "count from the end" protocol? How about slices?

It means the class defines a __getitem__ method. Like __call__, the semantics of that method, and the range of acceptable arguments, is out of scope of the ABC.

Then what use is this particular ABC? I assume you had a reason for wanting this when starting this thread but I still don't see what that is. The distinction between a Mapping and a Sequence is important because basic usage is different e.g.: for x in obj: print(obj[x]) # fine for Mapping but not for Sequence for n in range(len(obj)): print(obj[n]) # fine for Sequence but not for Mapping I can see situations where you might want to differentiate between these two in order to do something useful. I would still prefer not to write code that way myself and instead to simply document my expectation that e.g. a mapping is expected. I can see though why someone would want to use isinstance(obj, Mapping) in some situations. With Subscriptable I don't see a situation where the isinstance test is actually useful. I'm assuming here that your reason for wanting the Subscriptable ABC is to branch on isinstance but maybe that's not right. This discussion would be easier if you would give a clear example where this might be useful. -- Oscar

On Oct 1, 2019, at 15:59, Chris Angelico wrote:

...

On Wed, Oct 2, 2019 at 8:51 AM Greg Ewing wrote:

Chris Angelico wrote:

...

The question then would be: why is object() not hashable?

It's not hashable because it's supposed to be the ultimate base type for all other objects, and not every object is hashable.

It only seems odd if you're used to the idea that you get a bunch of default behaviours from object, including hashability. But if you want strict subtyping and also an ultimate base type, the base type has to include very little behaviour.

Should the default object be comparable? Or would there be a "Comparable" subtype? If object() can be compared for equality with another object(), then it'd be illogical to make it unhashable - there's no reason to deny hashing. If it can't be compared, why not? Why should the base type include THAT little behaviour?

This really is a practicality-vs.-purity question. Theoretically, it is wrong to include any behavior in your top type that is not supported by every other type in the system. Practically, it is useful to include behavior in your top type that is used by _most_ types in the system and would be a pain to reimplement over and over again, even if it isn’t supported by all types. Different languages solve part of that balance by having a lot of decorators, or mixin classes, or macros, or magic flags to the implementation. (Or you even could go Smalltalk style, and different environments will add different methods to Object.) Python does some of that too—functools.total_order, and some of the collections.abc types being their own mixins, etc. But it still gets tedious to include six mixins and four decorators on 90% of your types, so Python compromises purity and moves some of that stuff into object. So, object is hashable, and that means some types that are going to pass an is-a test for object are going to violate LSP. (Obviously that’s not the actual design history of Python—Guido didn’t decide to add object.__hash__ only after imagining the future of multiple inheritance and decorators, and comparing how the three different design choices would impact all the dict code people we’re going to write in future decades. But you get the idea.)

On Monday, September 30, 2019 at 12:24:30 AM UTC-4, Andrew Barnert via Python-ideas wrote:

On Sep 29, 2019, at 16:41, Steven D'Aprano javascript:> wrote:

...

...

On Sun, Sep 29, 2019 at 11:27:32AM +0100, Oscar Benjamin wrote:

That's the point that I would make as well. What can you do with an object that is only known to be Subscriptable?

I can subscript it. What did you expect the answer to be?

I think the idea here is that your type follows the “old sequence protocol”, where you can index it with 0, 1, … until you get an IndexError, without having to check __len__ first.

Since there are various bits of Python (including the iter function) that support this protocol, it’s odd that there’s no ABC for it. And it’s not just that it’s odd, it’s different from almost every other collection-related protocol in Python.

I think the only reasonable argument against this is that “old-style” means “legacy”, as in you shouldn’t be creating new classes like this. But I don’t think that argument is true. There are perfectly good use cases—like your infinite sequences—that are perfectly good old-style sequences and are not valid Sequences, and how else are you supposed to implement them? (Iteration is great, but sometimes random access matters.)

That's funny, I always thought of that as legacy. The iterator protocol has been a special case in so many proposals I've seen on this list. I think it's really ugly. Instead of collections.abc.OldStyleSequence, what do you think of adding something like InfiniteSequence to collections.abc instead? It's basically Sequence without __len__, __reversed__, or __count__. I don't see it getting much use though.

If I’m right about what you’re asking for, I think it’s a useful addition.

Of course the same protocol would accept all kinds of bizarre other things that support __getitem__ for different reasons, like the “I want to spell calling differently” thing, plus (somewhat less uselessly) typing._GenericAlias subclasses like typing.List. But I don’t think that’s a problem. Sure, it’s not ideal that your test would accept typing.List, but who’s going to pass the List pseudo-type to a function that clearly expects some kind of collection? If they get a different exception than they expected (a TypeError a few lines down, most likely), who cares? That seems like a consenting adults issue. _______________________________________________ Python-ideas mailing list -- python...@python.org javascript: To unsubscribe send an email to python-id...@python.org javascript: https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/LPQXOM... Code of Conduct: http://python.org/psf/codeofconduct/