[Python-ideas] Proposal: Use mypy syntax for function annotations

[Ed Kellett]

This proposal effectively negates any benefit to adding annotations to Python.

Annotations' only function was to provide unspecified information about
parameters and return values to other Python at runtime. That goes with the
acceptance of this proposal: It's still their sole function, but now we're
*not allowed to use them* to do that any more. Annotations alone are syntax
that corresponds to exactly no semantic meaning, and far from encouraging
developers to assign meaning to them, you're banning them from doing so.

In its current form, Python code can't even make use of the annotations that
are allowed: mypy's type system is not Python's, and without a specification
of how the static type checker's type system actually works, anything looking
at the annotations from the Python side would have to come up with its own way
to interpret them.

You could have picked *any* other way to communicate static typing information,
and it wouldn't neuter a feature so completely:

- use a @typechecked decorator to signify that a decorated function's
  annotations are for use as type information
- use a certain set of annotations to denote type information, without
  deprecating the use of annotations for other purposes
- parse a standardized docstring format for type information
- add new syntax for expressing type information without conflicting with
  annotations: def foo(int bar), for example

Static typing is what it is and I won't try to convince you that it's not
worth adding to the language, but it's not worth rendering a feature useless
when you could *not* render that feature useless and retain all the same
benefits.

Ed Kellett

On 13 August 2014 20:44, Guido van Rossum <guido at python.org> wrote:
> [There is no TL;DR other than the subject line. Please read the whole thing
> before replying. I do have an appendix with some motivations for adding type
> annotations at the end.]
>
> Yesterday afternoon I had an inspiring conversation with Bob Ippolito (man
> of many trades, author of simplejson) and Jukka Lehtosalo (author of mypy:
> http://mypy-lang.org/). Bob gave a talk at EuroPython about what Python can
> learn from Haskell (and other languages); yesterday he gave the same talk at
> Dropbox. The talk is online
> (https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad
> strokes comes down to three suggestions:
>
>   (a) Python should adopt mypy's syntax for function annotations
>   (b) Python's use of mutabe containers by default is wrong
>   (c) Python should adopt some kind of Abstract Data Types
>
> Proposals (b) and (c) don't feel particularly actionable (if you disagree
> please start a new thread, I'd be happy to discuss these further if there's
> interest) but proposal (a) feels right to me.
>
> So what is mypy?  It is a static type checker for Python written by Jukka
> for his Ph.D. thesis. The basic idea is that you add type annotations to
> your program using some custom syntax, and when running your program using
> the mypy interpreter, type errors will be found during compilation (i.e.,
> before the program starts running).
>
> The clever thing here is that the custom syntax is actually valid Python 3,
> using (mostly) function annotations: your annotated program will still run
> with the regular Python 3 interpreter. In the latter case there will be no
> type checking, and no runtime overhead, except to evaluate the function
> annotations (which are evaluated at function definition time but don't have
> any effect when the function is called).
>
> In fact, it is probably more useful to think of mypy as a heavy-duty linter
> than as a compiler or interpreter; leave the type checking to mypy, and the
> execution to Python. It is easy to integrate mypy into a continuous
> integration setup, for example.
>
> To read up on mypy's annotation syntax, please see the mypy-lang.org
> website. Here's just one complete example, to give a flavor:
>
>   from typing import List, Dict
>
>   def word_count(input: List[str]) -> Dict[str, int]:
>       result = {}  #type: Dict[str, int]
>       for line in input:
>           for word in line.split():
>               result[word] = result.get(word, 0) + 1
>       return result
>
> Note that the #type: comment is part of the mypy syntax; mypy uses comments
> to declare types in situations where no syntax is available -- although this
> particular line could also be written as follows:
>
>     result = Dict[str, int]()
>
> Either way the entire function is syntactically valid Python 3, and a
> suitable implementation of typing.py (containing class definitions for List
> and Dict, for example) can be written to make the program run correctly. One
> is provided as part of the mypy project.
>
> I should add that many of mypy's syntactic choices aren't actually new. The
> basis of many of its ideas go back at least a decade: I blogged about this
> topic in 2004 (http://www.artima.com/weblogs/viewpost.jsp?thread=85551 --
> see also the two followup posts linked from the top there).
>
> I'll emphasize once more that mypy's type checking happens in a separate
> pass: no type checking happens at run time (other than what the interpreter
> already does, like raising TypeError on expressions like 1+"1").
>
> There's a lot to this proposal, but I think it's possible to get a PEP
> written, accepted and implemented in time for Python 3.5, if people are
> supportive. I'll go briefly over some of the action items.
>
> (1) A change of direction for function annotations
>
> PEP 3107, which introduced function annotations, is intentional
> non-committal about how function annotations should be used. It lists a
> number of use cases, including but not limited to type checking. It also
> mentions some rejected proposals that would have standardized either a
> syntax for indicating types and/or a way for multiple frameworks to attach
> different annotations to the same function. AFAIK in practice there is
> little use of function annotations in mainstream code, and I propose a
> conscious change of course here by stating that annotations should be used
> to indicate types and to propose a standard notation for them.
>
> (We may have to have some backwards compatibility provision to avoid
> breaking code that currently uses annotations for some other purpose.
> Fortunately the only issue, at least initially, will be that when running
> mypy to type check such code it will produce complaints about the
> annotations; it will not affect how such code is executed by the Python
> interpreter. Nevertheless, it would be good to deprecate such alternative
> uses of annotations.)
>
> (2) A specification for what to add to Python 3.5
>
> There needs to be at least a rough consensus on the syntax for annotations,
> and the syntax must cover a large enough set of use cases to be useful. Mypy
> is still under development, and some of its features are still evolving
> (e.g. unions were only added a few weeks ago). It would be possible to argue
> endlessly about details of the notation, e.g. whether to use 'list' or
> 'List', what either of those means (is a duck-typed list-like type
> acceptable?) or how to declare and use type variables, and what to do with
> functions that have no annotations at all (mypy currently skips those
> completely).
>
> I am proposing that we adopt whatever mypy uses here, keeping discussion of
> the details (mostly) out of the PEP. The goal is to make it possible to add
> type checking annotations to 3rd party modules (and even to the stdlib)
> while allowing unaltered execution of the program by the (unmodified) Python
> 3.5 interpreter. The actual type checker will not be integrated with the
> Python interpreter, and it will not be checked into the CPython repository.
> The only thing that needs to be added to the stdlib is a copy of mypy's
> typing.py module. This module defines several dozen new classes (and a few
> decorators and other helpers) that can be used in expressing argument types.
> If you want to type-check your code you have to download and install mypy
> and run it separately.
>
> The curious thing here is that while standardizing a syntax for type
> annotations, we technically still won't be adopting standard rules for type
> checking. This is intentional. First of all, fully specifying all the type
> checking rules would make for a really long and boring PEP (a much better
> specification would probably be the mypy source code). Second, I think it's
> fine if the type checking algorithm evolves over time, or if variations
> emerge. The worst that can happen is that you consider your code correct but
> mypy disagrees; your code will still run.
>
> That said, I don't want to completely leave out any specification. I want
> the contents of the typing.py module to be specified in the PEP, so that it
> can be used with confidence. But whether mypy will complain about your
> particular form of duck typing doesn't have to be specified by the PEP.
> Perhaps as mypy evolves it will take options to tell it how to handle
> certain edge cases. Forks of mypy (or entirely different implementations of
> type checking based on the same annotation syntax) are also a possibility.
> Maybe in the distant future a version of Python will take a different
> stance, once we have more experience with how this works out in practice,
> but for Python 3.5 I want to restrict the scope of the upheaval.
>
> Appendix -- Why Add Type Annotations?
>
> The argument between proponents of static typing and dynamic typing has been
> going on for many decades. Neither side is all wrong or all right. Python
> has traditionally fallen in the camp of extremely dynamic typing, and this
> has worked well for most users, but there are definitely some areas where
> adding type annotations would help.
>
> - Editors (IDEs) can benefit from type annotations; they can call out
> obvious mistakes (like misspelled method names or inapplicable operations)
> and suggest possible method names. Anyone who has used IntelliJ or Xcode
> will recognize how powerful these features are, and type annotations will
> make such features more useful when editing Python source code.
>
> - Linters are an important tool for teams developing software. A linter
> doesn't replace a unittest, but can find certain types of errors better or
> quicker. The kind of type checking offered by mypy works much like a linter,
> and has similar benefits; but it can find problems that are beyond the
> capabilities of most linters.
>
> - Type annotations are useful for the human reader as well! Take the above
> word_count() example. How long would it have taken you to figure out the
> types of the argument and return value without annotations? Currently most
> people put the types in their docstrings; developing a standard notation for
> type annotations will reduce the amount of documentation that needs to be
> written, and running the type checker might find bugs in the documentation,
> too. Once a standard type annotation syntax is introduced, it should be
> simple to add support for this notation to documentation generators like
> Sphinx.
>
> - Refactoring. Bob's talk has a convincing example of how type annotations
> help in (manually) refactoring code. I also expect that certain automatic
> refactorings will benefit from type annotations -- imagine a tool like 2to3
> (but used for some other transformation) augmented by type annotations, so
> it will know whether e.g. x.keys() is referring to the keys of a dictionary
> or not.
>
> - Optimizers. I believe this is actually the least important application,
> certainly initially. Optimizers like PyPy or Pyston wouldn't be able to
> fully trust the type annotations, and they are better off using their
> current strategy of optimizing code based on the types actually observed at
> run time. But it's certainly feasible to imagine a future optimizer also
> taking type annotations into account.
>
> --
> --Guido "I need a new hobby" van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/