isinstance(instance, protocol) with protocol members without default values (PEP 544)
I hope to have been directed to the right place for my question from https://bugs.python.org/issue43512 I'm completely new to the Python dev process. Therefore, I ask for your indulgence if my approach is not in line with the process. I would like to collect opinions (especially from authors of PEP 544) if the CPython behavior described below should be regarded as a violation of PEP 544 or at least as a behavior that should be improved. The section "Subtyping relationships with other types" of PEP 544 states: "A concrete type X is a subtype of protocol P if and only if X implements all protocol members of P with compatible types. In other words, subtyping with respect to a protocol is always structural." In contrast to that, the behavior of CPython 3.9.2 for protocols with protocol members without default values is as follows: ``` from typing import Protocol class P(Protocol): protocol_member: str # no default value, but still a protocol member class X(P): # inherits P but does NOT implement protocol_member, since P did not provide a default value pass assert isinstance(X(), P) # violates the PEP 544 requirement cited above X().protocol_member # raises: AttributeError: 'X' object has no attribute 'protocol_member' ``` In this regard, it was argued that "At runtime, protocol classes will be simple ABCs." (PEP 544) But unfortunately, this is currently not the case. Actually, there is an extra metaclass for protocols, solely to provide an __instancecheck__. https://github.com/python/cpython/blob/3.9/Lib/typing.py#L1096 ``` class _ProtocolMeta(ABCMeta): # This metaclass is really unfortunate and exists only because of # the lack of __instancehook__. def __instancecheck__(cls, instance): # We need this method for situations where attributes are # assigned in __init__. if ((not getattr(cls, '_is_protocol', False) or _is_callable_members_only(cls)) and issubclass(instance.__class__, cls)): return True if cls._is_protocol: if all(hasattr(instance, attr) and # All *methods* can be blocked by setting them to None. (not callable(getattr(cls, attr, None)) or getattr(instance, attr) is not None) for attr in _get_protocol_attrs(cls)): return True return super().__instancecheck__(instance) ``` I am inclined to assess the behavior described above as a violation of PEP 544 and an incomplete implementation of `_ProtocolMeta`.
I believe you are saying: 1) `isinstance(X(), P)` succeeds when it should fail, because X doesn't have `protocol_member` defined. 2) Protocols are not ABCs at runtime when PEP 544 says they should be. I'm not one of the authors of PEP 544, so I can't give insight into why things are so. For 1: As noted by Ken Jin in the bug you linked, PEP 544 is concerned with Protocols as a static type checking concept, not a runtime concept. One reason is that this is impossible in some cases, such as due to type erasure. For example, even though `complex` implements `__float__`, it does not implement the SupportsFloat protocol, because complex's `__float__` signature differs from the SupportsFloat signature. This cannot be checked at runtime, because the type annotations are erased. Your example, with `P.protocol_member`, shows another case. Because `protocol_member` has no default value, it is never stored in `P`. (i.e. `hasattr(P, 'protocol_member') is False.) Therefore it cannot be checked at runtime. Static type checkers don't drop annotated values like this, so a static type checker can evaluate `isinstance(X(), P)` correctly. As PEP 544 says <https://www.python.org/dev/peps/pep-0544/#implementation-details>: "All structural subtyping checks will be performed by static type checkers, such as mypy [mypy]. No additional support for protocol validation will be provided at runtime." That said, I am a little confused as to why `isinstance(X(), P)` succeeds at all, because PEP 544 says "The default semantics is that isinstance() and issubclass() fail for protocol types." P should need to be decorated with `@runtime_checkable` for this to work. For 2: Protocols aren't literally instances of abc.ABCMeta, but they're still abstract base classes: they can't be instantiated and they define an interface that subclasses should implement. Saying they aren't ABCs at runtime is splitting hairs. -- Teddy On Thu, Mar 18, 2021 at 1:25 PM Paul Dest <paul.dest@web.de> wrote:
I hope to have been directed to the right place for my question from https://bugs.python.org/issue43512 I'm completely new to the Python dev process. Therefore, I ask for your indulgence if my approach is not in line with the process.
I would like to collect opinions (especially from authors of PEP 544) if the CPython behavior described below should be regarded as a violation of PEP 544 or at least as a behavior that should be improved.
The section "Subtyping relationships with other types" of PEP 544 states: "A concrete type X is a subtype of protocol P if and only if X implements all protocol members of P with compatible types. In other words, subtyping with respect to a protocol is always structural."
In contrast to that, the behavior of CPython 3.9.2 for protocols with protocol members without default values is as follows: ``` from typing import Protocol
class P(Protocol): protocol_member: str # no default value, but still a protocol member
class X(P): # inherits P but does NOT implement protocol_member, since P did not provide a default value pass
assert isinstance(X(), P) # violates the PEP 544 requirement cited above
X().protocol_member # raises: AttributeError: 'X' object has no attribute 'protocol_member' ```
In this regard, it was argued that "At runtime, protocol classes will be simple ABCs." (PEP 544)
But unfortunately, this is currently not the case. Actually, there is an extra metaclass for protocols, solely to provide an __instancecheck__. https://github.com/python/cpython/blob/3.9/Lib/typing.py#L1096
``` class _ProtocolMeta(ABCMeta): # This metaclass is really unfortunate and exists only because of # the lack of __instancehook__. def __instancecheck__(cls, instance): # We need this method for situations where attributes are # assigned in __init__. if ((not getattr(cls, '_is_protocol', False) or _is_callable_members_only(cls)) and issubclass(instance.__class__, cls)): return True if cls._is_protocol: if all(hasattr(instance, attr) and # All *methods* can be blocked by setting them to None. (not callable(getattr(cls, attr, None)) or getattr(instance, attr) is not None) for attr in _get_protocol_attrs(cls)): return True return super().__instancecheck__(instance) ```
I am inclined to assess the behavior described above as a violation of PEP 544 and an incomplete implementation of `_ProtocolMeta`. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: tsudol@google.com
1) Yes, I was saying `isinstance(X(), P)` succeeds when it should, because X doesn't have `protocol_member` defined. 2) No, I wasn't suggesting Protocols should be ABCs at runtime. My only intent was to play back the objection of Ken Jin to point 1 and my reaction to that. Unfortunately, my playback of Ken Jins objection was too abbreviated. His objection was based on the PEP 544 statement: "At runtime, protocol classes will be simple ABCs. There is no intent to provide sophisticated runtime instance and class checks against protocol classes. This would be difficult and error-prone and will contradict the logic of PEP 484." - My answer to this is: checking if a protocol member exists, is not a sophisticated, but the most basic check, and therefore should be done if not difficult or error-prone. As shown in the code, `_ProtocolMeta.__instancecheck__` already performs such kind of checks, additionally to the checks performed by `ABCMeta.__instancecheck__`. Regarding your (Teddy) explanations regarding 1: a) After your hint, I agree that it is surprising that `isinstance(X(), P)` in my code fragment succeeds at all. PEP 544 states: "A protocol can be used as a second argument in isinstance() and issubclass() only if it is explicitly opt-in by @runtime_checkable decorator." So, I think, the code fragment given in my initial message should already fail because of the missing @runtime_checkable decorator. But even with such a decorator, I think it should fail. But it does not fail: ``` from typing import Protocol, runtime_checkable @runtime_checkable class P(Protocol): protocol_member: str # no default value, but still a protocol member class X(P): # inherits P but does NOT implement protocol_member, since P did not provide a default value pass assert isinstance(X(), P) # should fail, but it doesn't X().protocol_member # raises: AttributeError: 'X' object has no attribute 'protocol_member' ``` b) As you (Teddy) pointed out, some properties of protocols cannot be checked at runtime, because the information is not available at runtime (erasing). Indeed, `protocol_member` is never stored as an attribute of `P`, as it has no default value resulting in `hasattr(P, 'protocol_member') is False`. But nevertheless `protocol_member` is stored in `P.__annotations__`, is available at runtime, and is actually returned by the current implementation of CPython's function `_get_protocol_attrs(cls)` in typing.py: https://github.com/python/cpython/blob/3.9/Lib/typing.py#L1048 It is one of the protocol attributes iterated over in the code of `_ProtocolMeta.__instancecheck__` shown in my first message: `for attr in _get_protocol_attrs(cls)`. For reasons that are incomprehensible for me, the current check ignores all protocol attributes which are not callable. I think the check should not be restricted to callables only. -- Paul
Correction: +NOT 1) Yes, I was saying `isinstance(X(), P)` succeeds when it should NOT, because X doesn't have `protocol_member` defined.
participants (2)
-
Paul Dest -
Teddy Sudol