Mailman 3 typing.override decorator - Typing-sig

20 May 2022

      Hi everyone!

Pyre has received several user requests over the last few months to support @override functionality in the Python type system. I’ve compiled some of details here to see what you all think (I hope the formatting works over the mailing list):

Motivation:

One core functionality of the type checker is to flag when refactors or changes break pre-existing semantic structures in the code, so users can identify and make fixes across their project without doing a manual audit of their code.

Unlike most other language type systems, Python does not provide a way to identify call sites that need to be changed to stay consistent when an overridden function API changes. This makes refactoring code dangerous and increases the cost of re-architectures and codemods.

Take for example, a simple inheritance structure:

class Base:
    def foo(self, input: int) -> int:
        return input

class Child(Base):
    def foo(self, input: int) -> int:
        return input

def callsite(input: Child) -> None:
    input.foo(1)

If an overridden method on the superclass is renamed or deleted, the type checker will only identify and flag call sites that deal with objects of the base class directly, but still happily accept any call sites dealing with subclasses.

class Base:
    def new_foo(self, input: int, flag: bool) -> int:
        # new semantics / side effects
        return input

class Child(Base):
    # this method now silently does something fundamentally different - defines a new function unrelated to what it used to override
    def foo(self, input: int) -> int:
        return input

def callsite(input: Child) -> None:
    input.foo(1)  # no indication this API changed

While this will run when executed, the type checker omits to mention the fact that a core piece of semantics have changed: the subclass method that was formerly overriding a superclass method is now, silently, defining a brand new method on the class and fooling call-sites into thinking nothing has changed with the old API. Explicitly marking a method as intending to override something in the parent would help us flag these problems in a complicated inheritance system.

We’ve seen runtime bug reports usually caused by an engineer renaming a parent class method and forgetting to rename that method in all child classes. This blows up at runtime and causes some user frustration because this pattern is caught in type systems of many other languages (see below).

Syntax & Semantics:

A method decorated with @typing.override will throw a type error if it is not actively overriding a compatible method in a (non-Protocol) parent class.

from typing import override

class Base:
    def foo(self) -> int:
        return 1

    def bar(self, x: str) -> str:
        return x

class Child(Base):
    @override
    def foo(self) -> int:
        return 2

    @override
    def bar(self, y: int) -> int: # type error: signature mismatch
        return y

    @override
    def baz() -> int:  # type error: no overridden method
        return 1

Note: Method decorators @staticmethod and @classmethod should always be applied last - in other words, @override should wrap the method more closely (should only matter for runtime).

from typing import override

class Base:
    @staticmethod
    def foo() -> int:
        return 1

    @classmethod
    def bar(cls, x: str) -> str:
        return x

class Child(Base):
    @staticmethod
    @override
    def foo() -> int:
        return 2

    @override
    def bar(self, x: str) -> str: # type error: signature mismatch
        return x

Adoption:

The proposed syntax and semantics above imply that the use of the @override decorator will be purely opt-in.

Alternatively, type checkers may want to offer an option for users to flag methods that are actively overriding a method in the parent class but are missing the decorator. There are a couple ways this may be possible:

  1.  Class Inheritance: Denote a class as “enforcing explicit overrides” (syntax suggestions welcome) and error on any subclass methods that override parent methods but are not decorated with @override.

This is what the PyPi runtime library (see below) provides via inheritance of a helper EnforceOverrides class. It would affect downstream dependencies that may subclass your logic but not be type checked together, and may be confusing

  1.  Everywhere: Make explicit annotation of @override a core expectation of well-annotated code, enforcing its usage in strictly typed modules across the board.

This is probably too invasive.

Precedent in other languages:

A non-exhaustive list of other languages that provide the same feature:

  *   Hack has <<__Override>><https://docs.hhvm.com/hack/attributes/predefined-attributes#__override>
  *   Java has @Override<https://docs.oracle.com/javase/tutorial/java/IandI/override.html>
  *   Scala has override<https://www.javatpoint.com/scala-method-overriding>
  *   Kotlin has override<https://kotlinlang.org/docs/inheritance.html#overriding-methods>
  *   Swift has override<https://docs.swift.org/swift-book/LanguageGuide/Inheritance.html#ID198>
  *   C++ has override<https://en.cppreference.com/w/cpp/language/override>
  *   JS/Flow discourage classes (in favor of functions)

Comparison to third-party runtime enforcement library:

There already exists a library<https://pypi.org/project/overrides/> that enforces @overrides (sic) and @final at runtime.

PEP 591<https://peps.python.org/pep-0591/> added a final decorator to the typing module. The override component of the runtime library is not supported statically at all, which has added some confusion around the mix/matched support where the runtime library is imported and used.

The ways in which the current static typing.override proposal differs from the runtime library are:

  1.  Proposed spelling is override rather than overrides
  2.  May not enforce an option where methods in classes inheriting from EnforceOverrides that are missing this decorator will throw type errors*

*The runtime library provides two motivation examples for EnforceOverrides unrelated to feature adoption that I do not find motivating for the type checking use case:

  1.  “A method that is added to a superclass is shadowed by an existing method with the same name in a subclass.”

The type checker will throw errors already here, unless the name, all parameters types and names (and arity), and return type are all perfectly compatible. That seems extremely unlikely unless the methods are in fact intended to do the same thing, which suggests this is not flagging a real error.

  1.  “A method of a superclass that is overridden by a subclass but the signature of the overridden method is incompatible with that of the inherited one.”

Similarly, the type checker will already throw incompatible override errors to flag this.

Looking for feedback on whether this would make a good addition to the typing module, as well as thoughts on some of the open questions regarding an “EnforceOverrides” equivalent or any other concerns.

Thanks!

Shannon

typing.override decorator

Shannon Zhu

Mehdi2277

Sergei Lebedev

Martin DeMello

Jia Chen

Eric Traut

Shannon Zhu

Teddy Sudol

Kristoffel Pirard

tags

participants (8)