<div dir="ltr"><div>Hi all,<br><br>After collecting suggestions in the previous discussion on python-dev <a href="https://mail.python.org/pipermail/python-dev/2017-March/thread.html#147629">https://mail.python.org/pipermail/python-dev/2017-March/thread.html#147629</a> and playing with implementation, here is an updated version of PEP 544.<br><br>--<br>Ivan<br><br><br></div>A link for those who don't like reading long e-mails: <a href="https://www.python.org/dev/peps/pep-0544/">https://www.python.org/dev/peps/pep-0544/</a><br><div><div><div><br>=========================<br><br>PEP: 544<br>Title: Protocols<br>Version: $Revision$<br>Last-Modified: $Date$<br>Author: Ivan Levkivskyi <<a href="mailto:levkivskyi@gmail.com">levkivskyi@gmail.com</a>>, Jukka Lehtosalo <<a href="mailto:jukka.lehtosalo@iki.fi">jukka.lehtosalo@iki.fi</a>>, Åukasz Langa <<a href="mailto:lukasz@langa.pl">lukasz@langa.pl</a>><br>Discussions-To: Python-Dev <<a href="mailto:python-dev@python.org">python-dev@python.org</a>><br>Status: Draft<br>Type: Standards Track<br>Content-Type: text/x-rst<br>Created: 05-Mar-2017<br>Python-Version: 3.7<br><br><br>Abstract<br>========<br><br>Type hints introduced in PEP 484 can be used to specify type metadata<br>for static type checkers and other third party tools. However, PEP 484<br>only specifies the semantics of *nominal* subtyping. In this PEP we specify<br>static and runtime semantics of protocol classes that will provide a support<br>for *structural* subtyping (static duck typing).<br><br><br>.. _rationale:<br><br>Rationale and Goals<br>===================<br><br>Currently, PEP 484 and the ``typing`` module [typing]_ define abstract<br>base classes for several common Python protocols such as ``Iterable`` and<br>``Sized``. The problem with them is that a class has to be explicitly marked<br>to support them, which is unpythonic and unlike what one would<br>normally do in idiomatic dynamically typed Python code. For example,<br>this conforms to PEP 484::<br><br> from typing import Sized, Iterable, Iterator<br><br> class Bucket(Sized, Iterable[int]):<br>   ...<br>   def __len__(self) -> int: ...<br>   def __iter__(self) -> Iterator[int]: ...<br><br>The same problem appears with user-defined ABCs: they must be explicitly<br>subclassed or registered. This is particularly difficult to do with library<br>types as the type objects may be hidden deep in the implementation<br>of the library. Also, extensive use of ABCs might impose additional<br>runtime costs.<br><br>The intention of this PEP is to solve all these problems<br>by allowing users to write the above code without explicit base classes in<br>the class definition, allowing ``Bucket`` to be implicitly considered<br>a subtype of both ``Sized`` and ``Iterable[int]`` by static type checkers<br>using structural [wiki-structural]_ subtyping::<br><br> from typing import Iterator, Iterable<br><br> class Bucket:<br>   ...<br>   def __len__(self) -> int: ...<br>   def __iter__(self) -> Iterator[int]: ...<br><br> def collect(items: Iterable[int]) -> int: ...<br> result: int = collect(Bucket())  # Passes type check<br><br>Note that ABCs in ``typing`` module already provide structural behavior<br>at runtime, ``isinstance(Bucket(), Iterable)`` returns ``True``.<br>The main goal of this proposal is to support such behavior statically.<br>The same functionality will be provided for user-defined protocols, as<br>specified below. The above code with a protocol class matches common Python<br>conventions much better. It is also automatically extensible and works<br>with additional, unrelated classes that happen to implement<br>the required protocol.<br><br><br>Nominal vs structural subtyping<br>-------------------------------<br><br>Structural subtyping is natural for Python programmers since it matches<br>the runtime semantics of duck typing: an object that has certain properties<br>is treated independently of its actual runtime class.<br>However, as discussed in PEP 483, both nominal and structural<br>subtyping have their strengths and weaknesses. Therefore, in this PEP we<br>*do not propose* to replace the nominal subtyping described by PEP 484 with<br>structural subtyping completely. Instead, protocol classes as specified in<br>this PEP complement normal classes, and users are free to choose<br>where to apply a particular solution. See section on `rejected`_ ideas at the<br>end of this PEP for additional motivation.<br><br><br>Non-goals<br>---------<br><br>At runtime, protocol classes will be simple ABCs. There is no intent to<br>provide sophisticated runtime instance and class checks against protocol<br>classes. This would be difficult and error-prone and will contradict the logic<br>of PEP 484. As well, following PEP 484 and PEP 526 we state that protocols are<br>**completely optional**:<br><br>* No runtime semantics will be imposed for variables or parameters annotated<br> with a protocol class.<br>* Any checks will be performed only by third-party type checkers and<br> other tools.<br>* Programmers are free to not use them even if they use type annotations.<br>* There is no intent to make protocols non-optional in the future.<br><br><br>Existing Approaches to Structural Subtyping<br>===========================================<br><br>Before describing the actual specification, we review and comment on existing<br>approaches related to structural subtyping in Python and other languages:<br><br>* ``zope.interface`` [zope-interfaces]_ was one of the first widely used<br> approaches to structural subtyping in Python. It is implemented by providing<br> special classes to distinguish interface classes from normal classes,<br> to mark interface attributes, and to explicitly declare implementation.<br> For example::<br><br>  from zope.interface import Interface, Attribute, implementer<br><br>  class IEmployee(Interface):<br><br>    name = Attribute("Name of employee")<br><br>    def do(work):<br>      """Do some work"""<br><br>  @implementer(IEmployee)<br>  class Employee:<br><br>    name = 'Anonymous'<br><br>    def do(self, work):<br>      return work.start()<br><br> Zope interfaces support various contracts and constraints for interface<br> classes. For example::<br><br>  from zope.interface import invariant<br><br>  def required_contact(obj):<br>    if not (obj.email or obj.phone):<br>      raise Exception("At least one contact info is required")<br><br>  class IPerson(Interface):<br><br>    name = Attribute("Name")<br>    email = Attribute("Email Address")<br>    phone = Attribute("Phone Number")<br><br>    invariant(required_contact)<br><br> Even more detailed invariants are supported. However, Zope interfaces rely<br> entirely on runtime validation. Such focus on runtime properties goes<br> beyond the scope of the current proposal, and static support for invariants<br> might be difficult to implement. However, the idea of marking an interface<br> class with a special base class is reasonable and easy to implement both<br> statically and at runtime.<br><br>* Python abstract base classes [abstract-classes]_ are the standard<br> library tool to provide some functionality similar to structural subtyping.<br> The drawback of this approach is the necessity to either subclass<br> the abstract class or register an implementation explicitly::<br><br>  from abc import ABC<br><br>  class MyTuple(ABC):<br>    pass<br><br>  MyTuple.register(tuple)<br><br>  assert issubclass(tuple, MyTuple)<br>  assert isinstance((), MyTuple)<br><br> As mentioned in the `rationale`_, we want to avoid such necessity, especially<br> in static context. However, in a runtime context, ABCs are good candidates for<br> protocol classes and they are already used extensively in<br> the ``typing`` module.<br><br>* Abstract classes defined in ``collections.abc`` module [collections-abc]_<br> are slightly more advanced since they implement a custom<br> ``__subclasshook__()`` method that allows runtime structural checks without<br> explicit registration::<br><br>  from collections.abc import Iterable<br><br>  class MyIterable:<br>    def __iter__(self):<br>      return []<br><br>  assert isinstance(MyIterable(), Iterable)<br><br> Such behavior seems to be a perfect fit for both runtime and static behavior<br> of protocols. As discussed in `rationale`_, we propose to add static support<br> for such behavior. In addition, to allow users to achieve such runtime<br> behavior for *user-defined* protocols a special ``@runtime`` decorator will<br> be provided, see detailed `discussion`_ below.<br><br>* TypeScript [typescript]_ provides support for user-defined classes and<br> interfaces. Explicit implementation declaration is not required and<br> structural subtyping is verified statically. For example::<br><br>  interface LabeledItem {<br>    label: string;<br>    size?: int;<br>  }<br><br>  function printLabel(obj: LabeledItem) {<br>    console.log(obj.label);<br>  }<br><br>  let myObj = {size: 10, label: "Size 10 Object"};<br>  printLabel(myObj);<br><br> Note that optional interface members are supported. Also, TypeScript<br> prohibits redundant members in implementations. While the idea of<br> optional members looks interesting, it would complicate this proposal and<br> it is not clear how useful it will be. Therefore it is proposed to postpone<br> this; see `rejected`_ ideas. In general, the idea of static protocol<br> checking without runtime implications looks reasonable, and basically<br> this proposal follows the same line.<br><br>* Go [golang]_ uses a more radical approach and makes interfaces the primary<br> way to provide type information. Also, assignments are used to explicitly<br> ensure implementation::<br><br>  type SomeInterface interface {<br>    SomeMethod() ([]byte, error)<br>  }<br><br>  if _, ok := someval.(SomeInterface); ok {<br>    fmt.Printf("value implements some interface")<br>  }<br><br> Both these ideas are questionable in the context of this proposal. See<br> the section on `rejected`_ ideas.<br><br><br>.. _specification:<br><br>Specification<br>=============<br><br>Terminology<br>-----------<br><br>We propose to use the term *protocols* for types supporting structural<br>subtyping. The reason is that the term *iterator protocol*,<br>for example, is widely understood in the community, and coming up with<br>a new term for this concept in a statically typed context would just create<br>confusion.<br><br>This has the drawback that the term *protocol* becomes overloaded with<br>two subtly different meanings: the first is the traditional, well-known but<br>slightly fuzzy concept of protocols such as iterator; the second is the more<br>explicitly defined concept of protocols in statically typed code.<br>The distinction is not important most of the time, and in other<br>cases we propose to just add a qualifier such as *protocol classes*<br>when referring to the static type concept.<br><br>If a class includes a protocol in its MRO, the class is called<br>an *explicit* subclass of the protocol. If a class is a structural subtype<br>of a protocol, it is said to implement the protocol and to be compatible<br>with a protocol. If a class is compatible with a protocol but the protocol<br>is not included in the MRO, the class is an *implicit* subtype<br>of the protocol. (Note that one can explicitly subclass a protocol and<br>still not implement it if a protocol attribute is set to ``None``<br>in the subclass, see Python [data-model]_ for details.)<br><br>The attributes (variables and methods) of a protocol that are mandatory<br>for other class in order to be considered a structural subtype are called<br>protocol members.<br><br><br>.. _definition:<br><br>Defining a protocol<br>-------------------<br><br>Protocols are defined by including a special new class ``typing.Protocol``<br>(an instance of ``abc.ABCMeta``) in the base classes list, typically<br>at the end of the list. Here is a simple example::<br><br> from typing import Protocol<br><br> class SupportsClose(Protocol):<br>   def close(self) -> None:<br>     ...<br><br>Now if one defines a class ``Resource`` with a ``close()`` method that has<br>a compatible signature, it would implicitly be a subtype of<br>``SupportsClose``, since the structural subtyping is used for<br>protocol types::<br><br> class Resource:<br>   ...<br>   def close(self) -> None:<br>     self.file.close()<br>     self.lock.release()<br><br>Apart from few restrictions explicitly mentioned below, protocol types can<br>be used in every context where a normal types can::<br><br> def close_all(things: Iterable[SupportsClose]) -> None:<br>   for t in things:<br>     t.close()<br><br> f = open('foo.txt')<br> r = Resource()<br> close_all([f, r])  # OK!<br> close_all([1])   # Error: 'int' has no 'close' method<br><br>Note that both the user-defined class ``Resource`` and the built-in<br>``IO`` type (the return type of ``open()``) are considered subtypes of<br>``SupportsClose``, because they provide a ``close()`` method with<br>a compatible type signature.<br><br><br>Protocol members<br>----------------<br><br>All methods defined in the protocol class body are protocol members, both<br>normal and decorated with ``@abstractmethod``. If any parameters of a<br>protocol method are not annotated, then their types are assumed to be ``Any``<br>(see PEP 484). Bodies of protocol methods are type checked.<br>An abstract method that should not be called via ``super()`` ought to raise<br>``NotImplementedError``. Example::<br><br> from typing import Protocol<br> from abc import abstractmethod<br><br> class Example(Protocol):<br>   def first(self) -> int:   # This is a protocol member<br>     return 42<br><br>   @abstractmethod<br>   def second(self) -> int:   # Method without a default implementation<br>     raise NotImplementedError<br><br>Static methods, class methods, and properties are equally allowed<br>in protocols.<br><br>To define a protocol variable, one can use PEP 526 variable<br>annotations in the class body. Additional attributes *only* defined in<br>the body of a method by assignment via ``self`` are not allowed. The rationale<br>for this is that the protocol class implementation is often not shared by<br>subtypes, so the interface should not depend on the default implementation.<br>Examples::<br><br> from typing import Protocol, List<br><br> class Template(Protocol):<br>   name: str     # This is a protocol member<br>   value: int = 0  # This one too (with default)<br><br>   def method(self) -> None:<br>     self.temp: List[int] = [] # Error in type checker<br><br> class Concrete:<br>   def __init__(self, name: str, value: int) -> None:<br>     <a href="http://self.name">self.name</a> = name<br>     self.value = value<br><br> var: Template = Concrete('value', 42)  # OK<br><br>To distinguish between protocol class variables and protocol instance<br>variables, the special ``ClassVar`` annotation should be used as specified<br>by PEP 526. By default, protocol variables as defined above are considered<br>readable and writable. To define a read-only protocol variable, one can use<br>an (abstract) property.<br><br><br>Explicitly declaring implementation<br>-----------------------------------<br><br>To explicitly declare that a certain class implements a given protocol,<br>it can be used as a regular base class. In this case a class could use<br>default implementations of protocol members. ``typing.Sequence`` is a good<br>example of a protocol with useful default methods. Static analysis tools are<br>expected to automatically detect that a class implements a given protocol.<br>So while it's possible to subclass a protocol explicitly, it's *not necessary*<br>to do so for the sake of type-checking.<br><br>The default implementations cannot be used if<br>the subtype relationship is implicit and only via structural<br>subtyping -- the semantics of inheritance is not changed. Examples::<br><br>  class PColor(Protocol):<br>    @abstractmethod<br>    def draw(self) -> str:<br>      ...<br>    def complex_method(self) -> int:<br>      # some complex code here<br><br>  class NiceColor(PColor):<br>    def draw(self) -> str:<br>      return "deep blue"<br><br>  class BadColor(PColor):<br>    def draw(self) -> str:<br>      return super().draw()  # Error, no default implementation<br><br>  class ImplicitColor:  # Note no 'PColor' base here<br>    def draw(self) -> str:<br>      return "probably gray"<br>    def comlex_method(self) -> int:<br>      # class needs to implement this<br><br>  nice: NiceColor<br>  another: ImplicitColor<br><br>  def represent(c: PColor) -> None:<br>    print(c.draw(), c.complex_method())<br><br>  represent(nice) # OK<br>  represent(another) # Also OK<br><br>Note that there is little difference between explicit and implicit<br>subtypes, the main benefit of explicit subclassing is to get some protocol<br>methods "for free". In addition, type checkers can statically verify that<br>the class actually implements the protocol correctly::<br><br>  class RGB(Protocol):<br>    rgb: Tuple[int, int, int]<br><br>    @abstractmethod<br>    def intensity(self) -> int:<br>      return 0<br><br>  class Point(RGB):<br>    def __init__(self, red: int, green: int, blue: str) -> None:<br>      self.rgb = red, green, blue  # Error, 'blue' must be 'int'<br><br>    # Type checker might warn that 'intensity' is not defined<br><br>A class can explicitly inherit from multiple protocols and also form normal<br>classes. In this case methods are resolved using normal MRO and a type checker<br>verifies that all subtyping are correct. The semantics of ``@abstractmethod``<br>is not changed, all of them must be implemented by an explicit subclass<br>before it can be instantiated.<br><br><br>Merging and extending protocols<br>-------------------------------<br><br>The general philosophy is that protocols are mostly like regular ABCs,<br>but a static type checker will handle them specially. Subclassing a protocol<br>class would not turn the subclass into a protocol unless it also has<br>``typing.Protocol`` as an explicit base class. Without this base, the class<br>is "downgraded" to a regular ABC that cannot be used with structural<br>subtyping. The rationale for this rule is that we don't want to accidentally<br>have some class act as a protocol just because one of its base classes<br>happens to be one. We still slightly prefer nominal subtyping over structural<br>subtyping in the static typing world.<br><br>A subprotocol can be defined by having *both* one or more protocols as<br>immediate base classes and also having ``typing.Protocol`` as an immediate<br>base class::<br><br> from typing import Sized, Protocol<br><br> class SizedAndClosable(Sized, Protocol):<br>   def close(self) -> None:<br>     ...<br><br>Now the protocol ``SizedAndClosable`` is a protocol with two methods,<br>``__len__`` and ``close``. If one omits ``Protocol`` in the base class list,<br>this would be a regular (non-protocol) class that must implement ``Sized``.<br>Alternatively, one can implement ``SizedAndClosable`` protocol by merging<br>the ``SupportsClose`` protocol from the example in the `definition`_ section<br>with ``typing.Sized``::<br><br> from typing import Sized<br><br> class SupportsClose(Protocol):<br>   def close(self) -> None:<br>     ...<br><br> class SizedAndClosable(Sized, SupportsClose, Protocol):<br>   pass<br><br>The two definitions of ``SizedAndClosable`` are equivalent.<br>Subclass relationships between protocols are not meaningful when<br>considering subtyping, since structural compatibility is<br>the criterion, not the MRO.<br><br>If ``Protocol`` is included in the base class list, all the other base classes<br>must be protocols. A protocol can't extend a regular class, see `rejected`_<br>ideas for rationale. Note that rules around explicit subclassing are different<br>from regular ABCs, where abstractness is simply defined by having at least one<br>abstract method being unimplemented. Protocol classes must be marked<br>*explicitly*.<br><br><br>Generic protocols<br>-----------------<br><br>Generic protocols are important. For example, ``SupportsAbs``, ``Iterable``<br>and ``Iterator`` are generic protocols. They are defined similar to normal<br>non-protocol generic types::<br><br> class Iterable(Protocol[T]):<br>   @abstractmethod<br>   def __iter__(self) -> Iterator[T]:<br>     ...<br><br>``Protocol[T, S, ...]`` is allowed as a shorthand for<br>``Protocol, Generic[T, S, ...]``.<br><br>User-defined generic protocols support explicitly declared variance.<br>Type checkers will warn if the inferred variance is different from<br>the declared variance. Examples::<br><br> T = TypeVar('T')<br> T_co = TypeVar('T_co', covariant=True)<br> T_contra = TypeVar('T_contra', contravariant=True)<br><br> class Box(Protocol[T_co]):<br>   def content(self) -> T_co:<br>     ...<br><br> box: Box[float]<br> second_box: Box[int]<br> box = second_box  # This is OK due to the covariance of 'Box'.<br><br> class Sender(Protocol[T_contra]):<br>   def send(self, data: T_contra) -> int:<br>     ...<br><br> sender: Sender[float]<br> new_sender: Sender[int]<br> new_sender = sender  # OK, 'Sender' is contravariant.<br><br> class Proto(Protocol[T]):<br>   attr: T  # this class is invariant, since it has a mutable attribute<br><br> var: Proto[float]<br> another_var: Proto[int]<br> var = another_var  # Error! 'Proto[float]' is incompatible with 'Proto[int]'.<br><br>Note that unlike nominal classes, de-facto covariant protocols cannot be<br>declared as invariant, since this can break transitivity of subtyping<br>(see `rejected`_ ideas for details). For example::<br><br> T = TypeVar('T')<br><br> class AnotherBox(Protocol[T]):  # Error, this protocol is covariant in T,<br>   def content(self) -> T:   # not invariant.<br>     ...<br><br><br>Recursive protocols<br>-------------------<br><br>Recursive protocols are also supported. Forward references to the protocol<br>class names can be given as strings as specified by PEP 484. Recursive<br>protocols are useful for representing self-referential data structures<br>like trees in an abstract fashion::<br><br> class Traversable(Protocol):<br>   def leaves(self) -> Iterable['Traversable']:<br>     ...<br><br>Note that for recursive protocols, a class is considered a subtype of<br>the protocol in situations where the decision depends on itself.<br>Continuing the previous example::<br><br> class SimpleTree:<br>   def leaves(self) -> List['SimpleTree']:<br>     ...<br><br> root: Traversable = SimpleTree()  # OK<br><br> class Tree(Generic[T]):<br>   def leaves(self) -> List['Tree[T]']:<br>     ...<br><br> def walk(graph: Traversable) -> None:<br>   ...<br> tree: Tree[float] = Tree()<br> walk(tree)  # OK, 'Tree[float]' is a subtype of 'Traversable'<br><br><br>Using Protocols<br>===============<br><br>Subtyping relationships with other types<br>----------------------------------------<br><br>Protocols cannot be instantiated, so there are no values whose<br>runtime type is a protocol. For variables and parameters with protocol types,<br>subtyping relationships are subject to the following rules:<br><br>* A protocol is never a subtype of a concrete type.<br>* A concrete type ``X`` is a subtype of protocol ``P``<br> if and only if ``X`` implements all protocol members of ``P`` with<br> compatible types. In other words, subtyping with respect to a protocol is<br> always structural.<br>* A protocol ``P1`` is a subtype of another protocol ``P2`` if ``P1`` defines<br> all protocol members of ``P2`` with compatible types.<br><br>Generic protocol types follow the same rules of variance as non-protocol<br>types. Protocol types can be used in all contexts where any other types<br>can be used, such as in ``Union``, ``ClassVar``, type variables bounds, etc.<br>Generic protocols follow the rules for generic abstract classes, except for<br>using structural compatibility instead of compatibility defined by<br>inheritance relationships.<br><br><br>Unions and intersections of protocols<br>-------------------------------------<br><br>``Union`` of protocol classes behaves the same way as for non-protocol<br>classes. For example::<br><br> from typing import Union, Optional, Protocol<br><br> class Exitable(Protocol):<br>   def exit(self) -> int:<br>     ...<br> class Quittable(Protocol):<br>   def quit(self) -> Optional[int]:<br>     ...<br><br> def finish(task: Union[Exitable, Quittable]) -> int:<br>   ...<br> class DefaultJob:<br>   ...<br>   def quit(self) -> int:<br>     return 0<br> finish(DefaultJob()) # OK<br><br>One can use multiple inheritance to define an intersection of protocols.<br>Example::<br><br> from typing import Sequence, Hashable<br><br> class HashableFloats(Sequence[float], Hashable, Protocol):<br>   pass<br><br> def cached_func(args: HashableFloats) -> float:<br>   ...<br> cached_func((1, 2, 3)) # OK, tuple is both hashable and sequence<br><br>If this will prove to be a widely used scenario, then a special<br>intersection type construct could be added in future as specified by PEP 483,<br>see `rejected`_ ideas for more details.<br><br><br>``Type[]`` with protocols<br>-------------------------<br><br>Variables and parameters annotated with ``Type[Proto]`` accept only concrete<br>(non-protocol) subtypes of ``Proto``. The main reason for this is to allow<br>instantiation of parameters with such type. For example::<br><br> class Proto(Protocol):<br>   @abstractmethod<br>   def meth(self) -> int:<br>     ...<br> class Concrete:<br>   def meth(self) -> int:<br>     return 42<br><br> def fun(cls: Type[Proto]) -> int:<br>   return cls().meth() # OK<br> fun(Proto)        # Error<br> fun(Concrete)      # OK<br><br>The same rule applies to variables::<br><br> var: Type[Proto]<br> var = Proto   # Error<br> var = Concrete # OK<br> var().meth()  # OK<br><br>Assigning an ABC or a protocol class to a variable is allowed if it is<br>not explicitly typed, and such assignment creates a type alias.<br>For normal (non-abstract) classes, the behavior of ``Type[]`` is<br>not changed.<br><br><br>``NewType()`` and type aliases<br>------------------------------<br><br>Protocols are essentially anonymous. To emphasize this point, static type<br>checkers might refuse protocol classes inside ``NewType()`` to avoid an<br>illusion that a distinct type is provided::<br><br> from typing import NewType, Protocol, Iterator<br><br> class Id(Protocol):<br>   code: int<br>   secrets: Iterator[bytes]<br><br> UserId = NewType('UserId', Id)  # Error, can't provide distinct type<br><br>In contrast, type aliases are fully supported, including generic type<br>aliases::<br><br> from typing import TypeVar, Reversible, Iterable, Sized<br><br> T = TypeVar('T')<br> class SizedIterable(Iterable[T], Sized, Protocol):<br>   pass<br> CompatReversible = Union[Reversible[T], SizedIterable[T]]<br><br><br>.. _discussion:<br><br>``@runtime`` decorator and narrowing types by ``isinstance()``<br>--------------------------------------------------------------<br><br>The default semantics is that ``isinstance()`` and ``issubclass()`` fail<br>for protocol types. This is in the spirit of duck typing -- protocols<br>basically would be used to model duck typing statically, not explicitly<br>at runtime.<br><br>However, it should be possible for protocol types to implement custom<br>instance and class checks when this makes sense, similar to how ``Iterable``<br>and other ABCs in ``collections.abc`` and ``typing`` already do it,<br>but this is limited to non-generic and unsubscripted generic protocols<br>(``Iterable`` is statically equivalent to ``Iterable[Any]`).<br>The ``typing`` module will define a special ``@runtime`` class decorator<br>that provides the same semantics for class and instance checks as for<br>``collections.abc`` classes, essentially making them "runtime protocols"::<br><br> from typing import runtime, Protocol<br><br> @runtime<br> class Closable(Protocol):<br>   def close(self):<br>     ...<br><br> assert isinstance(open('some/file'), Closable)<br><br>Static type checkers will understand ``isinstance(x, Proto)`` and<br>``issubclass(C, Proto)`` for protocols defined with this decorator (as they<br>already do for ``Iterable`` etc.). Static type checkers will narrow types<br>after such checks by the type erased ``Proto`` (i.e. with all variables<br>having type ``Any`` and all methods having type ``Callable[..., Any]``).<br>Note that ``isinstance(x, Proto[int])`` etc. will always fail in agreement<br>with PEP 484. Examples::<br><br> from typing import Iterable, Iterator, Sequence<br><br> def process(items: Iterable[int]) -> None:<br>   if isinstance(items, Iterator):<br>     # 'items' has type 'Iterator[int]' here<br>   elif isinstance(items, Sequence[int]):<br>     # Error! Can't use 'isinstance()' with subscripted protocols<br><br>Note that instance checks are not 100% reliable statically, this is why<br>this behavior is opt-in, see section on `rejected`_ ideas for examples.<br><br><br>Using Protocols in Python 2.7 - 3.5<br>===================================<br><br>Variable annotation syntax was added in Python 3.6, so that the syntax<br>for defining protocol variables proposed in `specification`_ section can't<br>be used if support for earlier versions is needed. To define these<br>in a manner compatible with older versions of Python one can use properties.<br>Properties can be settable and/or abstract if needed::<br><br> class Foo(Protocol):<br>   @property<br>   def c(self) -> int:<br>     return 42     # Default value can be provided for property...<br><br>   @abstractproperty<br>   def d(self) -> int:  # ... or it can be abstract<br>     return 0<br><br>Also function type comments can be used as per PEP 484 (for example<br>to provide compatibility with Python 2). The ``typing`` module changes<br>proposed in this PEP will also be backported to earlier versions via the<br>backport currently available on PyPI.<br><br><br>Runtime Implementation of Protocol Classes<br>==========================================<br><br>Implementation details<br>----------------------<br><br>The runtime implementation could be done in pure Python without any<br>effects on the core interpreter and standard library except in the<br>``typing`` module, and a minor update to ``collections.abc``:<br><br>* Define class ``typing.Protocol`` similar to ``typing.Generic``.<br>* Implement metaclass functionality to detect whether a class is<br> a protocol or not. Add a class attribute ``_is_protocol = True``<br> if that is the case. Verify that a protocol class only has protocol<br> base classes in the MRO (except for object).<br>* Implement ``@runtime`` that allows ``__subclasshook__()`` performing<br> structural instance and subclass checks as in ``collections.abc`` classes.<br>* All structural subtyping checks will be performed by static type checkers,<br> such as ``mypy`` [mypy]_. No additional support for protocol validation will<br> be provided at runtime.<br>* Classes ``Mapping``, ``MutableMapping``, ``Sequence``, and<br> ``MutableSequence`` in ``collections.abc`` module will support structural<br> instance and subclass checks (like e.g. ``collections.abc.Iterable``).<br><br><br>Changes in the typing module<br>----------------------------<br><br>The following classes in ``typing`` module will be protocols:<br><br>* ``Callable``<br>* ``Awaitable``<br>* ``Iterable``, ``Iterator``<br>* ``AsyncIterable``, ``AsyncIterator``<br>* ``Hashable``<br>* ``Sized``<br>* ``Container``<br>* ``Collection``<br>* ``Reversible``<br>* ``Sequence``, ``MutableSequence``<br>* ``Mapping``, ``MutableMapping``<br>* ``ContextManager``, ``AsyncContextManager``<br>* ``SupportsAbs`` (and other ``Supports*`` classes)<br><br>Most of these classes are small and conceptually simple. It is easy to see<br>what are the methods these protocols implement, and immediately recognize<br>the corresponding runtime protocol counterpart.<br>Practically, few changes will be needed in ``typing`` since some of these<br>classes already behave the necessary way at runtime. Most of these will need<br>to be updated only in the corresponding ``typeshed`` stubs [typeshed]_.<br><br>All other concrete generic classes such as ``List``, ``Set``, ``IO``,<br>``Deque``, etc are sufficiently complex that it makes sense to keep<br>them non-protocols (i.e. require code to be explicit about them). Also, it is<br>too easy to leave some methods unimplemented by accident, and explicitly<br>marking the subclass relationship allows type checkers to pinpoint the missing<br>implementations.<br><br><br>Introspection<br>-------------<br><br>The existing class introspection machinery (``dir``, ``__annotations__`` etc)<br>can be used with protocols. In addition, all introspection tools implemented<br>in the ``typing`` module will support protocols. Since all attributes need<br>to be defined in the class body based on this proposal, protocol classes will<br>have even better perspective for introspection than regular classes where<br>attributes can be defined implicitly -- protocol attributes can't be<br>initialized in ways that are not visible to introspection<br>(using ``setattr()``, assignment via ``self``, etc.). Still, some things like<br>types of attributes will not be visible at runtime in Python 3.5 and earlier,<br>but this looks like a reasonable limitation.<br><br>There will be only limited support of ``isinstance()`` and ``issubclass()``<br>as discussed above (these will *always* fail with ``TypeError`` for<br>subscripted generic protocols, since a reliable answer could not be given<br>at runtime in this case). But together with other introspection tools this<br>give a reasonable perspective for runtime type checking tools.<br><br><br>.. _rejected:<br><br>Rejected/Postponed Ideas<br>========================<br><br>The ideas in this section were previously discussed in [several]_<br>[discussions]_ [elsewhere]_.<br><br>Make every class a protocol by default<br>--------------------------------------<br><br>Some languages such as Go make structural subtyping the only or the primary<br>form of subtyping. We could achieve a similar result by making all classes<br>protocols by default (or even always). However we believe that it is better<br>to require classes to be explicitly marked as protocols, for the following<br>reasons:<br><br>* Protocols don't have some properties of regular classes. In particular,<br> ``isinstance()``, as defined for normal classes, is based on the nominal<br> hierarchy. In order to make everything a protocol by default, and have<br> ``isinstance()`` work would require changing its semantics,<br> which won't happen.<br>* Protocol classes should generally not have many method implementations,<br> as they describe an interface, not an implementation.<br> Most classes have many method implementations, making them bad protocol<br> classes.<br>* Experience suggests that many classes are not practical as protocols anyway,<br> mainly because their interfaces are too large, complex or<br> implementation-oriented (for example, they may include de facto<br> private attributes and methods without a ``__`` prefix).<br>* Most actually useful protocols in existing Python code seem to be implicit.<br> The ABCs in ``typing`` and ``collections.abc`` are rather an exception, but<br> even they are recent additions to Python and most programmers<br> do not use them yet.<br>* Many built-in functions only accept concrete instances of ``int``<br> (and subclass instances), and similarly for other built-in classes. Making<br> ``int`` a structural type wouldn't be safe without major changes to the<br> Python runtime, which won't happen.<br><br><br>Protocols subclassing normal classes<br>------------------------------------<br><br>The main rationale to prohibit this is to preserve transitivity of subtyping,<br>consider this example::<br><br> from typing import Protocol<br><br> class Base:<br>   attr: str<br><br> class Proto(Base, Protocol):<br>   def meth(self) -> int:<br>     ...<br><br> class C:<br>   attr: str<br>   def meth(self) -> int:<br>     return 0<br><br>Now, ``C`` is a subtype of ``Proto``, and ``Proto`` is a subtype of ``Base``.<br>But ``C`` cannot be a subtype of ``Base`` (since the latter is not<br>a protocol). This situation would be really weird. In addition, there is<br>an ambiguity about whether attributes of ``Base`` should become protocol<br>members of ``Proto``.<br><br><br>Support optional protocol members<br>---------------------------------<br><br>We can come up with examples where it would be handy to be able to say<br>that a method or data attribute does not need to be present in a class<br>implementing a protocol, but if it is present, it must conform to a specific<br>signature or type. One could use a ``hasattr()`` check to determine whether<br>they can use the attribute on a particular instance.<br><br>Languages such as TypeScript have similar features and<br>apparently they are pretty commonly used. The current realistic potential<br>use cases for protocols in Python don't require these. In the interest<br>of simplicity, we propose to not support optional methods or attributes.<br>We can always revisit this later if there is an actual need.<br><br><br>Allow only protocol methods and force use of getters and setters<br>----------------------------------------------------------------<br><br>One could argue that protocols typically only define methods, but not<br>variables. However, using getters and setters in cases where only a<br>simple variable is needed would be quite unpythonic. Moreover, the widespread<br>use of properties (that often act as type validators) in large code bases<br>is partially due to previous absence of static type checkers for Python,<br>the problem that PEP 484 and this PEP are aiming to solve. For example::<br><br> # without static types<br><br> class MyClass:<br>   @property<br>   def my_attr(self):<br>     return self._my_attr<br>   @my_attr.setter<br>   def my_attr(self, value):<br>     if not isinstance(value, int):<br>       raise ValidationError("An integer expected for my_attr")<br>     self._my_attr = value<br><br> # with static types<br><br> class MyClass:<br>   my_attr: int<br><br><br>Support non-protocol members<br>----------------------------<br><br>There was an idea to make some methods "non-protocol" (i.e. not necessary<br>to implement, and inherited in explicit subclassing), but it was rejected,<br>since this complicates things. For example, consider this situation::<br><br> class Proto(Protocol):<br>   @abstractmethod<br>   def first(self) -> int:<br>     raise NotImplementedError<br>   def second(self) -> int:<br>     return self.first() + 1<br><br> def fun(arg: Proto) -> None:<br>   arg.second()<br><br>The question is should this be an error? We think most people would expect<br>this to be valid. Therefore, to be on the safe side, we need to require both<br>methods to be implemented in implicit subclasses. In addition, if one looks<br>at definitions in ``collections.abc``, there are very few methods that could<br>be considered "non-protocol". Therefore, it was decided to not introduce<br>"non-protocol" methods.<br><br>There is only one downside to this: it will require some boilerplate for<br>implicit subtypes of ``Mapping`` and few other "large" protocols. But, this<br>applies to few "built-in" protocols (like ``Mapping`` and ``Sequence``) and<br>people are already subclassing them. Also, such style is discouraged for<br>user-defined protocols. It is recommended to create compact protocols and<br>combine them.<br><br><br>Make protocols interoperable with other approaches<br>--------------------------------------------------<br><br>The protocols as described here are basically a minimal extension to<br>the existing concept of ABCs. We argue that this is the way they should<br>be understood, instead of as something that *replaces* Zope interfaces,<br>for example. Attempting such interoperabilities will significantly<br>complicate both the concept and the implementation.<br><br>On the other hand, Zope interfaces are conceptually a superset of protocols<br>defined here, but using an incompatible syntax to define them,<br>because before PEP 526 there was no straightforward way to annotate attributes.<br>In the 3.6+ world, ``zope.interface`` might potentially adopt the ``Protocol``<br>syntax. In this case, type checkers could be taught to recognize interfaces<br>as protocols and make simple structural checks with respect to them.<br><br><br>Use assignments to check explicitly that a class implements a protocol<br>----------------------------------------------------------------------<br><br>In the Go language the explicit checks for implementation are performed<br>via dummy assignments [golang]_. Such a way is also possible with the<br>current proposal. Example::<br><br> class A:<br>   def __len__(self) -> float:<br>     return ...<br><br> _: Sized = A()  # Error: A.__len__ doesn't conform to 'Sized'<br>         # (Incompatible return type 'float')<br><br>This approach moves the check away from<br>the class definition and it almost requires a comment as otherwise<br>the code probably would not make any sense to an average reader<br>-- it looks like dead code. Besides, in the simplest form it requires one<br>to construct an instance of ``A``, which could be problematic if this requires<br>accessing or allocating some resources such as files or sockets.<br>We could work around the latter by using a cast, for example, but then<br>the code would be ugly. Therefore we discourage the use of this pattern.<br><br><br>Support ``isinstance()`` checks by default<br>------------------------------------------<br><br>The problem with this is instance checks could be unreliable, except for<br>situations where there is a common signature convention such as ``Iterable``.<br>For example::<br><br> class P(Protocol):<br>   def common_method_name(self, x: int) -> int: ...<br><br> class X:<br>   <a bunch of methods><br>   def common_method_name(self) -> None: ... # Note different signature<br><br> def do_stuff(o: Union[P, X]) -> int:<br>   if isinstance(o, P):<br>     return o.common_method_name(1)  # oops, what if it's an X instance?<br><br>Another potentially problematic case is assignment of attributes<br>*after* instantiation::<br><br> class P(Protocol):<br>   x: int<br><br> class C:<br>   def initialize(self) -> None:<br>     self.x = 0<br><br> c = C()<br> isinstance(c1, P)  # False<br> c.initialize()<br> isinstance(c, P)  # True<br><br> def f(x: Union[P, int]) -> None:<br>   if isinstance(x, P):<br>     # static type of x is P here<br>     ...<br>   else:<br>     # type of x is "int" here?<br>     print(x + 1)<br><br> f(C())  # oops<br><br>We argue that requiring an explicit class decorator would be better, since<br>one can then attach warnings about problems like this in the documentation.<br>The user would be able to evaluate whether the benefits outweigh<br>the potential for confusion for each protocol and explicitly opt in -- but<br>the default behavior would be safer. Finally, it will be easy to make this<br>behavior default if necessary, while it might be problematic to make it opt-in<br>after being default.<br><br><br>Provide a special intersection type construct<br>---------------------------------------------<br><br>There was an idea to allow ``Proto = All[Proto1, Proto2, ...]`` as a shorthand<br>for::<br><br> class Proto(Proto1, Proto2, ..., Protocol):<br>   pass<br><br>However, it is not yet clear how popular/useful it will be and implementing<br>this in type checkers for non-protocol classes could be difficult. Finally, it<br>will be very easy to add this later if needed.<br><br><br>Prohibit explicit subclassing of protocols by non-protocols<br>-----------------------------------------------------------<br><br>This was rejected for the following reasons:<br><br>* Backward compatibility: People are already using ABCs, including generic<br> ABCs from ``typing`` module. If we prohibit explicit subclassing of these<br> ABCs, then quite a lot of code will break.<br><br>* Convenience: There are existing protocol-like ABCs (that will be turned<br> into protocols) that have many useful "mix-in" (non-abstract) methods.<br> For example in the case of ``Sequence`` one only needs to implement<br> ``__getitem__`` and ``__len__`` in an explicit subclass, and one gets<br> ``__iter__``, ``__contains__``, ``__reversed__``, ``index``, and ``count``<br> for free.<br><br>* Explicit subclassing makes it explicit that a class implements a particular<br> protocol, making subtyping relationships easier to see.<br><br>* Type checkers can warn about missing protocol members or members with<br> incompatible types more easily, without having to use hacks like dummy<br> assignments discussed above in this section.<br><br>* Explicit subclassing makes it possible to force a class to be considered<br> a subtype of a protocol (by using ``# type: ignore`` together with an<br> explicit base class) when it is not strictly compatible, such as when<br> it has an unsafe override.<br><br><br>Covariant subtyping of mutable attributes<br>-----------------------------------------<br><br>Rejected because covariant subtyping of mutable attributes is not safe.<br>Consider this example::<br><br> class P(Protocol):<br>   x: float<br><br> def f(arg: P) -> None:<br>   arg.x = 0.42<br><br> class C:<br>   x: int<br><br> c = C()<br> f(c)  # Would typecheck if covariant subtyping<br>    # of mutable attributes were allowed<br> c.x >> 1  # But this fails at runtime<br><br>It was initially proposed to allow this for practical reasons, but it was<br>subsequently rejected, since this may mask some hard to spot bugs.<br><br><br>Overriding inferred variance of protocol classes<br>------------------------------------------------<br><br>It was proposed to allow declaring protocols as invariant if they are actually<br>covariant or contravariant (as it is possible for nominal classes, see PEP 484).<br>However, it was decided not to do this because of several downsides:<br><br>* Declared protocol invariance breaks transitivity of sub-typing. Consider<br> this situation::<br><br>  T = TypeVar('T')<br><br>  class P(Protocol[T]):  # Declared as invariant<br>    def meth(self) -> T:<br>      ...<br>  class C:<br>    def meth(self) -> float:<br>      ...<br>  class D(C):<br>    def meth(self) -> int:<br>      ...<br><br> Now we have that ``D`` is a subtype of ``C``, and ``C`` is a subtype of<br> ``P[float]``. But ``D`` is *not* a subtype of ``P[float]`` since ``D``<br> implements ``P[int]``, and ``P`` is invariant. There is a possibility<br> to "cure" this by looking for protocol implementations in MROs but this<br> will be too complex in a general case, and this "cure" requires abandoning<br> simple idea of purely structural subtyping for protocols.<br><br>* Subtyping checks will always require type inference for protocols. In the<br> above example a user may complain: "Why did you infer ``P[int]`` for<br> my ``D``? It implements ``P[float]``!". Normally, inference can be overruled<br> by an explicit annotation, but here this will require explicit subclassing,<br> defeating the purpose of using protocols.<br><br>* Allowing overriding variance will make impossible more detailed error<br> messages in type checkers citing particular conflicts in member<br> type signatures.<br><br>* Finally, explicit is better than implicit in this case. Requiring user to<br> declare correct variance will simplify understanding the code and will avoid<br> unexpected errors at the point of use.<br><br><br>Support adapters and adaptation<br>-------------------------------<br><br>Adaptation was proposed by PEP 246 (rejected) and is supported by<br>``zope.interface``, see <a href="https://docs.zope.org/zope.interface/adapter.html">https://docs.zope.org/zope.interface/adapter.html</a>.<br>Adapters is quite an advanced concept, and PEP 484 supports unions and<br>generic aliases that can be used instead of adapters. This can be illustrated<br>with an example of ``Iterable`` protocol, there is another way of supporting<br>iteration by providing ``__getitem__`` and ``__len__``. If a function<br>supports both this way and the now standard ``__iter__`` method, then it could<br>be annotated by a union type::<br><br> class OldIterable(Sized, Protocol[T]):<br>   def __getitem__(self, item: int) -> T: ...<br><br> CompatIterable = Union[Iterable[T], OldIterable[T]]<br><br> class A:<br>   def __iter__(self) -> Iterator[str]: ...<br> class B:<br>   def __len__(self) -> int: ...<br>   def __getitem__(self, item: int) -> str: ...<br><br> def iterate(it: CompatIterable[str]) -> None:<br>   ...<br><br> iterate(A())  # OK<br> iterate(B())  # OK<br><br>Since there is a reasonable alternative for such cases with existing tooling,<br>it is therefore proposed not to include adaptation in this PEP.<br><br><br>Backwards Compatibility<br>=======================<br><br>This PEP is almost fully backwards compatible. Few collection classes such as<br>``Sequence`` and ``Mapping`` will be turned into runtime protocols, therefore<br>results of ``isinstance()`` checks are going to change in some edge cases.<br>For example, a class that implements the ``Sequence`` protocol but does not<br>explicitly inherit from ``Sequence`` currently returns ``False`` in<br>corresponding instance and class checks. With this PEP implemented, such<br>checks will return ``True``.<br><br><br>Implementation<br>==============<br><br>A working implementation of this PEP for ``mypy`` type checker is found on<br>GitHub repo at <a href="https://github.com/ilevkivskyi/mypy/tree/protocols">https://github.com/ilevkivskyi/mypy/tree/protocols</a>,<br>corresponding ``typeshed`` stubs for more flavor are found at<br><a href="https://github.com/ilevkivskyi/typeshed/tree/protocols">https://github.com/ilevkivskyi/typeshed/tree/protocols</a>. Installation steps::<br><br> git clone --recurse-submodules <a href="https://github.com/ilevkivskyi/mypy/">https://github.com/ilevkivskyi/mypy/</a><br> cd mypy && git checkout protocols && cd typeshed<br> git remote add proto <a href="https://github.com/ilevkivskyi/typeshed">https://github.com/ilevkivskyi/typeshed</a><br> git fetch proto && git checkout proto/protocols<br> cd .. && git add typeshed && sudo python3 -m pip install -U .<br><br>The runtime implementation of protocols in ``typing`` module is<br>found at <a href="https://github.com/ilevkivskyi/typehinting/tree/protocols">https://github.com/ilevkivskyi/typehinting/tree/protocols</a>.<br>The version of ``collections.abc`` with structural behavior for mappings and<br>sequences is found at <a href="https://github.com/ilevkivskyi/cpython/tree/protocols">https://github.com/ilevkivskyi/cpython/tree/protocols</a>.<br><br><br>References<br>==========<br><br>.. [typing]<br>  <a href="https://docs.python.org/3/library/typing.html">https://docs.python.org/3/library/typing.html</a><br><br>.. [wiki-structural]<br>  <a href="https://en.wikipedia.org/wiki/Structural_type_system">https://en.wikipedia.org/wiki/Structural_type_system</a><br><br>.. [zope-interfaces]<br>  <a href="https://zopeinterface.readthedocs.io/en/latest/">https://zopeinterface.readthedocs.io/en/latest/</a><br><br>.. [abstract-classes]<br>  <a href="https://docs.python.org/3/library/abc.html">https://docs.python.org/3/library/abc.html</a><br><br>.. [collections-abc]<br>  <a href="https://docs.python.org/3/library/collections.abc.html">https://docs.python.org/3/library/collections.abc.html</a><br><br>.. [typescript]<br>  <a href="https://www.typescriptlang.org/docs/handbook/interfaces.html">https://www.typescriptlang.org/docs/handbook/interfaces.html</a><br><br>.. [golang]<br>  <a href="https://golang.org/doc/effective_go.html#interfaces_and_types">https://golang.org/doc/effective_go.html#interfaces_and_types</a><br><br>.. [data-model]<br>  <a href="https://docs.python.org/3/reference/datamodel.html#special-method-names">https://docs.python.org/3/reference/datamodel.html#special-method-names</a><br><br>.. [typeshed]<br>  <a href="https://github.com/python/typeshed/">https://github.com/python/typeshed/</a><br><br>.. [mypy]<br>  <a href="http://github.com/python/mypy/">http://github.com/python/mypy/</a><br><br>.. [several]<br>  <a href="https://mail.python.org/pipermail/python-ideas/2015-September/thread.html#35859">https://mail.python.org/pipermail/python-ideas/2015-September/thread.html#35859</a><br><br>.. [discussions]<br>  <a href="https://github.com/python/typing/issues/11">https://github.com/python/typing/issues/11</a><br><br>.. [elsewhere]<br>  <a href="https://github.com/python/peps/pull/224">https://github.com/python/peps/pull/224</a><br><br><br>Copyright<br>=========<br><br>This document has been placed in the public domain.<br></div></div></div></div>