[Python-ideas] Proposal to extend PEP 484 (gradual typing) to support Python 2.7

[Guido van Rossum]

At Dropbox we're trying to be good citizens and we're working towards
introducing gradual typing (PEP 484) into our Python code bases (several
million lines of code). However, that code base is mostly still Python 2.7
and we believe that we should introduce gradual typing first and start
working on conversion to Python 3 second (since having static types in the
code can help a big refactoring like that).

Since Python 2 doesn't support function annotations we've had to look for
alternatives. We considered stub files, a magic codec, docstrings, and
additional `# type:` comments. In the end we decided that `# type:`
comments are the most robust approach. We've experimented a fair amount
with this and we have a proposal for a standard.

The proposal is very simple. Consider the following function with Python 3
annotations:

    def embezzle(self, account: str, funds: int = 1000000, *fake_receipts:
str) -> None:
        """Embezzle funds from account using fake receipts."""
        <code goes here>

An equivalent way to write this in Python 2 is the following:

    def embezzle(self, account, funds=1000000, *fake_receipts):
        # type: (str, int, *str) -> None
        """Embezzle funds from account using fake receipts."""
        <code goes here>

There are a few details to discuss:

- Every argument must be accounted for, except 'self' (for instance
methods) or 'cls' (for class methods). Also the return type is mandatory.
If in Python 3 you would omit some argument or the return type, the Python
2 notation should use 'Any'.

- If you're using names defined in the typing module, you must still import
them! (There's a backport on PyPI.)

- For `*args` and `**kwds`, put 1 or 2 starts in front of the corresponding
type annotation. As with Python 3 annotations, the annotation here denotes
the type of the individual argument values, not of the tuple/dict that you
receive as the special argument value 'args' or 'kwds'.

- The entire annotation must be one line. (However, see
https://github.com/JukkaL/mypy/issues/1102.)

We would like to propose this as a standard (either to be added to PEP 484
or as a new PEP) rather than making it a "proprietary" extension to mypy
only, so that others in a similar situation can also benefit.

A brief discussion of the considered alternatives:

- Stub files: this would complicate the analysis in mypy quite a bit,
because it would have to parse both the .py file and the .pyi file and
somehow combine the information gathered from both, and for each function
it would have to use the types from the stub file to type-check the body of
the function in the .py file. This would require a lot of additional
plumbing. And if we were using Python 3 we would want to use in-line
annotations anyway.

- A magic codec was implemented over a year ago (
https://github.com/JukkaL/mypy/tree/master/mypy/codec) but after using it
for a bit we didn't like it much. It slows down imports, it requires a `#
coding: mypy` declaration, it would conflict with pyxl (
https://github.com/dropbox/pyxl), things go horribly wrong when the codec
isn't installed and registered, other tools would be confused by the Python
3 syntax in Python 2 source code, and because of the way the codec was
implemented the Python interpreter would occasionally spit out confusing
error messages showing the codec's output (which is pretty bare-bones).

- While there are existing conventions for specifying types in docstrings,
we haven't been using any of these conventions (at least not consistently,
nor at an appreciable scale), and they are much more verbose if all you
want is adding argument annotations. We're working on a tool that
automatically adds type annotations[1], and such a tool would be
complicated by the need to integrate the generated annotations into
existing docstrings (which, in our code base, unfortunately are wildly
incongruous in their conventions).

- Finally, the proposed comment syntax is easy to mechanically translate
into standard Python 3 function annotations once we're ready to let go of
Python 2.7.

__________
[1] I have a prototype of such a tool, i mplemented as a 2to3 fixer. It's a
bit over 200 lines. It's not very interesting yet, since it sets the types
of nearly all arguments to 'Any'. We're considering building a much more
advanced version that tries to guess much better argument types using some
form of whole-program analysis. I've heard that Facebook's Hack project got
a lot of mileage out of such a tool. I don't yet know how to write it yet
-- possibly we could use a variant of mypy's type inference engine, or
alternatively we might be able to use something like Jedi (
https://github.com/davidhalter/jedi).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20160108/087f0949/attachment-0001.html>