[Python-ideas] Proposal: Use mypy syntax for function annotations

[Steven D'Aprano]

On Sat, Aug 23, 2014 at 09:44:02AM -0400, Antoine Pitrou wrote:
> Le 23/08/2014 01:13, Steven D'Aprano a écrit :
> >On Fri, Aug 22, 2014 at 10:31:18PM -0400, Antoine Pitrou wrote:
> >
> >>>>Python has one of the most powerful and user-friendly function call
> >>>>syntaxes around, why reinvent something clearly inferior and alien?
> >>>
> >>>I wouldn't say it is alien. abc[xyz] is clearly Python syntax.
> >>
> >>But it is completely uncommon for anything else than subscripting and
> >>indexing containers.
> >
> >And function call syntax is completely uncommon for anything else than
> >calling functions (including types and methods). One way or the other,
> >we're adding a new use, type declarations.
> 
> It's not a new use. A type class is a class, and calling it is just 
> instantiating that class. There's nothing new here. If you think that's 
> a bit "meta", it's no different than e.g. higher-order functions.

There's no instantiation during *static* analysis, because the code 
hasn't run yet.

I don't think it's productive to argue about what's new and what isn't. 
I think it is more important to discuss what appears to me to be a 
problem with your suggestion that annotations use call syntax rather 
than subscript syntax. Here's an annotation:

    param:Sequence[int]

or as you prefer:

    param:Sequence(int)

There's two problems with the later, as I see it. Of course, during 
static analysis, we can use any syntax we like, but Python annotations 
are also evaluated at runtime.

(1) Sequence is an ABC, and like all abstract classes, you're not 
supposed to be able to instantiate it:

py> Sequence()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class Sequence with abstract 
methods __getitem__, __len__

For your proposal to be accepted, that would have to change. I think 
that is a problem. At the very least, it's a backwards-incompatibility, 
even within Python 3.x.

(2) Even if we allow instantiating Sequence, and returning a type 
(rather than an instance of Sequence), I don't think that Sequence 
should accept arguments that concrete subclasses would accept:

py> list(int)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'type' object is not iterable

So we would have to allow list() to accept type arguments, and return a 
type, as well as the usual iterable arguments. That strikes me as messy, 
ugly and confusing. I would much prefer keeping call syntax on a type to 
instantiate the type, so that list(obj) either fails or returns a list. 
list[int] can return anything needed, it doesn't have to be an instance 
of list.


> "The brackets looks better" is a misguided argument, the same way the 
> idea that "print" was cuter as a non-parenthetical statement (and that 
> we wouldn't need the power of a regular function call; eventually we 
> *did* need it) was a misguided argument.

You're entitled to your opinion, but in my opinion, trying to avoid 
confusing, hard to read syntax is never misguided. Overloading call 
syntax to do two things (instantiate types, and type annotations), and 
placing such calls in the function parameter list which already has 
parens, risks creating hard to read, ugly code. I think that using [] 
helps the annotations stand out, and I think that allowing the full 
function call syntax complete with keyword-only arguments inside type 
annotations is a case of YAGNI as well as confusing and hard to read. 
That's my opinion; yours may differ.

In any case, I think this is a minor point, and would rather not get 
into arguments about aesthetics. Ultimately, Guido will decide which one 
he prefers the look of.


> >The
> >annotations themselves will remain arbitrary Python expressions, so if
> >you really need a complex type declaration, you can create a helper and
> >call it with whatever signature you like.[1]
> 
> I don't think you understand what this discussion is about. It isn't 
> only about annotations (they are only the channel used to convey the 
> information), it is about standardizing a typing system in the stdlib - 
> and, therefore, accross the Python community.

I think you've got that exactly backwards. Guido's original post seemed 
clear to me that this proposal was *not* about adding a typing system to 
CPython, at least not now, but *only* about standardizing on the syntax 
used. A couple of quotes from his original post:

    [GvR]
    The actual type checker will not be integrated with the Python
    interpreter, and it will not be checked into the CPython repository. 
    The only thing that needs to be added to the stdlib is a copy of
    mypy's typing.py module. [...]

    The curious thing here is that while standardizing a syntax for type
    annotations, we technically still won't be adopting standard rules for
    type checking. This is intentional.


> >It is unclear to me just how powerful the type
> >language will be, but surely we don't expect it to evaluate arbitrarily
> >complex Python expressions at compile time? Do we?
> 
> Why wouldn't it? Ideally, "compile time" is just the import of the module.
> (yes, some libraries are ill-behaved on import; they break pydoc and the 
> like; I'd say it's their problem)

When I say compile-time, I mean when the module is compiled to byte 
code, before it is executed or imported.


> >- how powerful do we expect the type system to be?
> 
> This is a bad question to ask. That's like asking "how powerful does a 
> function or decorator need to be?" The entire point of devising new 
> language or library tools is to enable use cases that we *don't know 
> about yet*.

Decorators are an excellent example. Have you noticed how limited the 
syntax for decorators are? Probably not, because you can do a lot with 
decorators even though the syntax is deliberately limited. But limited 
it is. For instance, you can't use decorator syntax with an index 
lookup:

py> @decorators[0]
  File "<stdin>", line 1
    @decorators[0]
               ^
SyntaxError: invalid syntax

or more than one call:

py> @decorate(x)(y)
  File "<stdin>", line 1
    @decorate(x)(y)
                ^
SyntaxError: invalid syntax

(Both examples in 3.3; 3.4 may differ.)

Does this limit the power of decorators? No, of course not. The sorts of 
things decorators can do is only lightly constrained by the restrictions 
on syntax, and I believe that the same will apply to type annotations.


[...]
> >- and how much of that power needs to be expressed using
> >   function annotations?
> >
> >The second question is critical, because there are alternatives to
> >function annotations: decorators, docstring annotations, and external
> >stub files.
> 
> Why would those other channels use a type description syntax different 
> from the one used in function annotations? That would be crazy.

Because function annotations are limited to a single expression, while 
the others are not.

You're talking about me wanting to limit the power of the static type 
system, but I'm not really. I'm just wanting to limit how much clutter 
ends up inside the function parameter list. Not every arbitrarily 
complex type description needs to be written as a single expression 
inside the parameter list. Static analysis tools can make use of:

* function annotations;
* class definitions outside of the function (e.g. ABCs);
* decorators;
* docstring annotations;
* stub files;
* type inference on variables;
* even inline comments (as mypy does)

Guido's proposal is only about the first, and possibly by implication 
the second. The rest aren't going away, nor are they being standardised 
yet. There's plenty of opportunity for future tools to develop.


> You seem to think that a type system should simply be some kind of 
> textual description. It's not. It's a set of objects (or classes) with 
> some behaviour attached to them; that's why it uses Python syntax. 
> Because it *is* Python code.

That may be true about *runtime* typing, but it is not true for *static* 
typing since that occurs before the Python code runs.

You're absolutely correct that there are other uses for type annotations 
apart from static type checking, and if you go back over my previous 
posts you'll see I've made a number of references to dynamic runtime 
behaviour. Ethan even (politely) told me off for forgetting the static 
part of the proposal, when he thought I was emphasising the runtime 
implications too much. For you now to think I've forgotten the runtime 
implications is ironic :-)


-- 
Steven