Creating a Protocol from a normal class
Hi everyone, One of the rejected ideas in PEP 544 is being able to define a Protocol that inherits from a normal class (https://www.python.org/dev/peps/pep-0544/#protocols-subclassing-normal-class...). I've been thinking about a potential new feature that could accomplish more or less the same thing without running into the transitivity issue described in PEP 544. I'm sure there are some considerations I've missed, but I'd love to hear what you all think. In short, we'd introduce a +`.from()` method to typing.Protocol that takes in a class and returns a new Protocol with the same interface as that class. ``` from typing import Protocol class SomeClass: def some_method(self) -> None: pass class SomeClassProtocol(Protocol.from(SomeClass)): def another_method(self) -> None: ... ``` Static type checkers would in this case determine that SomeClassProtocol is a Protocol with two methods (some_method and another_method). If SomeClass and SomeClassProtocol both define a method with the same name, the one defined in SomeClassProtocol would take precedence. The next thing to think about would be: - Should the result of Protocol.from(SomeClass) only include SomeClass's methods, or should it include methods and attributes? - By default, I think it should probably include methods and attributes. There could possibly be an optional argument to Protocol.from() that toggles whether attributes are included, something like: `Protocol.fom(SomeClass, include_attrs=False)` - Should it include only "public" members, or should it also include `_` and `__` prefixed names? - My inclination would be that it should only include public members. Some other things to consider that I'm still thinking about: - What would SomeClassProtocol look like at runtime? I suppose Protocol.from(SomeClass) could introspect SomeClass at runtime and essentially build a protocol class that shares has SomeClass's interface? - Would this play nicely with generics? (I think/hope so?) One general use case (there are certainly others that I'm not thinking of at the moment) for this is when annotating "self" for a mixin class (https://mypy.readthedocs.io/en/stable/more_types.html#mixin-classes). It's at best cumbersome to have to write a protocol that matches an existing class and keep them in sync, and it's even more problematic if the protocol needs to match a class from another library. If this idea seems useful enough to move forward, I'd be happy to help write a PEP for it and work on a mypy implementation (this would be my first time writing a PEP or contributing to mypy). Looking forward to hearing your thoughts! Thanks, James
I see some utility in this idea, but I'm not sure if it's sufficient to merit a new facility. The use cases it supports strike me as relatively rare. Maintaining separate protocol class definitions doesn't seem to be that much of a burden for these infrequent use cases. But I'm interested in hearing from others. A few additional things to consider. Since all real classes subclass from `object`, I presume that the derived protocol would not automatically include all of the methods and attributes defined in `object` unless they were explicitly defined by the template class or one of its parent classes (other than object). That includes methods like `__eq__`, `__hash__`, and `__repr__`. The `.from` form looks odd to me. I think it would be more consistent with existing forms (e.g. `NewType` or `NamedTuple`) to adopt this form: ```python SomeClassProtocol = Protocol("SomeClassProtocol", SomeClass) ``` It's not clear to me how this would work for generic classes. Would the type variables used in the template class be adopted implicitly by the protocol class that's derived from the template? Does the same apply to type variables referenced by methods and non-method attributes? There's no precedent for implicit type variables anywhere else in the type system today, so this strikes me as a potential problem. Maybe the type variables need to be explicit, like this? ```python SomeClassProtocol = Protocol("SomeClassProtocol", SomeClass[_K, _V]) ``` How would type checkers differentiate between class variables and instance variables? PEP 526 provides a way to differentiate using `ClassVar`. Protocol class definitions tend to use `ClassVar` properly, but I rarely see it used in non-protocol class definitions. What about attributes that have no type annotation (e.g. the statement `self.foo = 3` within a method body)? Would they be included in the derived protocol class? -Eric -- Eric Traut Contributor to Pylance & Pyright Microsoft Corp.
On Fri, May 21, 2021 at 11:08 PM Eric Traut <eric@traut.com> wrote:
[snip] Since all real classes subclass from `object`, I presume that the derived protocol would not automatically include all of the methods and attributes defined in `object` unless they were explicitly defined by the template class or one of its parent classes (other than object). That includes methods like `__eq__`, `__hash__`, and `__repr__`.
That seems reasonable to me.
The `.from` form looks odd to me. I think it would be more consistent with existing forms (e.g. `NewType` or `NamedTuple`) to adopt this form:
```python SomeClassProtocol = Protocol("SomeClassProtocol", SomeClass) ```
I like this idea. I think the runtime behavior might need to be more in the spirit of NamedTuple, since I think it's pretty important to be able to define a protocol that derives from SomeClassProtocol and adds new fields. Would that be confusing?
It's not clear to me how this would work for generic classes. Would the type variables used in the template class be adopted implicitly by the protocol class that's derived from the template? Does the same apply to type variables referenced by methods and non-method attributes? There's no precedent for implicit type variables anywhere else in the type system today, so this strikes me as a potential problem. Maybe the type variables need to be explicit, like this?
```python SomeClassProtocol = Protocol("SomeClassProtocol", SomeClass[_K, _V]) ```
Agreed, the type variables should be explicit. FWIW TypeScript requires this too.
How would type checkers differentiate between class variables and instance variables? PEP 526 provides a way to differentiate using `ClassVar`. Protocol class definitions tend to use `ClassVar` properly, but I rarely see it used in non-protocol class definitions.
Is this much different from how type checkers handle class variables in general? That is, does the type checker ever infer a variable as a class variable if it isn't explicitly annotated with ClassVar? I'd assume the answer to that is "no," in which case I think this new form should only interpret a variable as a class variable if it's annotated with ClassVar. What about attributes that have no type annotation (e.g. the statement
`self.foo = 3` within a method body)? Would they be included in the derived protocol class?
Assuming we're talking about this in the context of __init__, then ideally yes, but that would tremendously complicate the runtime behavior... Would it make sense to have this new form behave somewhat like NewType in that it returns something that behaves like an identity function when called (or maybe raises an error in some cases) but at the same time allows deriving from it? e.g.: ```python SomeClassProtocol = Protocol("SomeClassProtocol", SomeClass) class DerivedSomeClassProtocol(SomeClassProtocol): new_field: int ``` Thanks, James
A similar proposal, just FYI https://github.com/python/mypy/issues/7894 On Fri, May 21, 2021 at 11:41 PM James Perretta <perretta.james@gmail.com> wrote:
Hi everyone,
One of the rejected ideas in PEP 544 is being able to define a Protocol that inherits from a normal class ( https://www.python.org/dev/peps/pep-0544/#protocols-subclassing-normal-class...). I've been thinking about a potential new feature that could accomplish more or less the same thing without running into the transitivity issue described in PEP 544. I'm sure there are some considerations I've missed, but I'd love to hear what you all think.
In short, we'd introduce a +`.from()` method to typing.Protocol that takes in a class and returns a new Protocol with the same interface as that class. ``` from typing import Protocol
class SomeClass: def some_method(self) -> None: pass
class SomeClassProtocol(Protocol.from(SomeClass)): def another_method(self) -> None: ... ```
Static type checkers would in this case determine that SomeClassProtocol is a Protocol with two methods (some_method and another_method). If SomeClass and SomeClassProtocol both define a method with the same name, the one defined in SomeClassProtocol would take precedence.
The next thing to think about would be: - Should the result of Protocol.from(SomeClass) only include SomeClass's methods, or should it include methods and attributes? - By default, I think it should probably include methods and attributes. There could possibly be an optional argument to Protocol.from() that toggles whether attributes are included, something like: `Protocol.fom(SomeClass, include_attrs=False)` - Should it include only "public" members, or should it also include `_` and `__` prefixed names? - My inclination would be that it should only include public members.
Some other things to consider that I'm still thinking about: - What would SomeClassProtocol look like at runtime? I suppose Protocol.from(SomeClass) could introspect SomeClass at runtime and essentially build a protocol class that shares has SomeClass's interface? - Would this play nicely with generics? (I think/hope so?)
One general use case (there are certainly others that I'm not thinking of at the moment) for this is when annotating "self" for a mixin class ( https://mypy.readthedocs.io/en/stable/more_types.html#mixin-classes). It's at best cumbersome to have to write a protocol that matches an existing class and keep them in sync, and it's even more problematic if the protocol needs to match a class from another library.
If this idea seems useful enough to move forward, I'd be happy to help write a PEP for it and work on a mypy implementation (this would be my first time writing a PEP or contributing to mypy). Looking forward to hearing your thoughts!
Thanks, James _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: epsilonmichael@gmail.com
On Sat, May 22, 2021 at 1:59 AM Michael Mitchell <epsilonmichael@gmail.com> wrote:
A similar proposal, just FYI https://github.com/python/mypy/issues/7894
Thanks for pointing that out. : ) I'm admittedly not quite sure what Ivan was referring to about extra dependencies and "fine-grained mode." Thanks, James
What Ivan meant was that this might be complex to implement in mypy. On Mon, May 24, 2021 at 7:22 AM James Perretta <perretta.james@gmail.com> wrote:
On Sat, May 22, 2021 at 1:59 AM Michael Mitchell <epsilonmichael@gmail.com> wrote:
A similar proposal, just FYI https://github.com/python/mypy/issues/7894
Thanks for pointing that out. : ) I'm admittedly not quite sure what Ivan was referring to about extra dependencies and "fine-grained mode."
Thanks, James
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
participants (4)
-
Eric Traut
-
Guido van Rossum
-
James Perretta
-
Michael Mitchell