On Tue, Aug 30, 2016 at 07:15:55PM -0700, Guido van Rossum wrote:

On Tue, Aug 30, 2016 at 6:00 PM, Steven D'Aprano <steve@pearwood.info> wrote:

...

On Tue, Aug 30, 2016 at 02:20:26PM -0700, Guido van Rossum wrote:

...

I'm happy to present PEP 526 for your collective review:

Are you hoping to get this in before 3.6 beta? Because I'm not sure I can give this much attention before then, but I really want to.

Yes I am hoping for that. Unlike PEP 484, this PEP is forward-looking (more like PEP 492, async/await), and the sooner we can get it in the sooner people who want to use it won't have to worry about supporting older Python versions. (And am I ever looking forward to the day when Python 3.5 counts as "older". :-)

Indeed :-) Okay, I'll bump it up my list of priorities. (Sleep? Who needs that? :-) -- Steve

On Tue, Aug 30, 2016 at 7:44 PM, Jack Diederich <jackdied@gmail.com> wrote:

+0. We should try and be consistent even if this is a thing I don't want. And trust me, I don't!

No problem. You won't have to!

That said, as long as pro-mypy people are willing to make everyone else pay a mypy reading tax for code let's try and reduce the cognitive burden.

* Duplicate type annotations should be a syntax error. Duplicate annotations aren't possible in functions so that wasn't an issue in 484. 526 makes some things syntax errors and some things runtime errors (for good reason -- function bodies aren't evaluated right away). Double-annotating a variable is something we can figure out at compile time and doing the double annotating is non-sensical so we should error on it because we can.

Actually I'm not so sure that double-annotating is always nonsensical. In the mypy tracker we're seeing some requests for type *inference* that allows a variable to be given another type later, e.g. x = 'abc' test_func(x) x = 42 another_test_func(x) Maybe there's a use for explicit annotations too. I would rather not get in the way of letting type checkers decide such semantics.

* Dissallowing annotations on global and nonlocal Agreed, allowing it would be confusing because it would either be a re-definition or a hard to read annotation-at-a-distance.

* Where __annotations__ live It is strange to allow modules.__annotations__ and MyClass.__annotations__ but not myfunc.__annotations__ (or more in line with existing function implementations a myfunc.__code__.co_annotations). If we know enough from the syntax parse to have func.__code__.co_varnames be known then we should try to do that with annotations. Let's raise a SyntaxError for function body annotations that conflict with same-named variables that are annotated in the function signature as well.

But myfunc.__annotations__ already exists -- PEP 3107 puts the signature annotations there. The problem with co_annotations is that annotations are evaluated (they can be quite complex expressions, e.g. Optional[Tuple[int, int, some_mod.SomeClass]]), while co_varnames is just a list of strings. And code objects must be immutable. The issue with rejecting duplicate annotations so sternly is the same as for the previous bullet.

I did C++ for years before I did Python and wrote C++ in many languages (including Python). So ideally I'm -1000 on all this stuff for cultural reasons -- if you let a C++ person add types they will for false comfort. But again, I'm +0 on this specific proposal because we have already gone down the garden path.

As long as you run mypy the comfort shouldn't be false. (But your starting with C++ before Python explains a lot. :-)

-Jack

-- --Guido van Rossum (python.org/~guido)

On 31 August 2016 at 13:37, Jack Diederich <jackdied@gmail.com> wrote:

On Tue, Aug 30, 2016 at 11:03 PM, Guido van Rossum <guido@python.org> wrote:

...

But myfunc.__annotations__ already exists -- PEP 3107 puts the signature annotations there. The problem with co_annotations is that annotations are evaluated (they can be quite complex expressions, e.g. Optional[Tuple[int, int, some_mod.SomeClass]]), while co_varnames is just a list of strings. And code objects must be immutable. The issue with rejecting duplicate annotations so sternly is the same as for the previous bullet.

If we disallow re-assignment of types as a syntax error then the conflict with myfunc.__annotations__ goes away for vars that share a name with the function arguments. The fact that variables with types can't be known until the function body executes a particular line is .. I'm not sure how to deal with that. For modules and classes you can assert that the body at the top indent level has been executed. For functions you can only assert that it has been parsed. So myfunc.__annotations__ could say that the type has a definition but only later know what the definition is.

What if we included local variable annotations in func.__annotations__ as cells, like the entries in func.__closure__? We could also use that as a micro-optimisation technique: once the type annotation cell is populated, CPython would just use it, rather than re-evaluating the local variable type annotation expression every time the function is called. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 31 August 2016 at 15:40, Guido van Rossum <guido@python.org> wrote:

On Tuesday, August 30, 2016, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

What if we included local variable annotations in func.__annotations__ as cells, like the entries in func.__closure__?

We could also use that as a micro-optimisation technique: once the type annotation cell is populated, CPython would just use it, rather than re-evaluating the local variable type annotation expression every time the function is called.

But what runtime use have the annotations on locals? They are not part of any inspectable interface. I don't want to spend any effort on them at runtime. (Just the bit that they are treated as locals.)

I guess as long as they're included somewhere in the AST for the function body, I don't mind if the translation to bytecode throws them away - that's essentially saying that a function level type annotation is effectively interpreted as if it was: if False: __annotations__[<varname>] = <annotation> So the code generator will pick up syntax errors during normal execution, but not runtime errors (since the expression never actually gets evaluated). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, Sep 1, 2016 at 5:46 PM, Guido van Rossum <guido@python.org> wrote:

On Thu, Sep 1, 2016 at 6:11 AM, Koos Zevenhoven <k7hoven@gmail.com> wrote:

...

While a large amount of Python programmers may not be interested in type hinting local variables inside functions, I can see other potential benefits in this.

IOW, PEP 3157 is not dead yet. Indeed.

PEP 3157? Is that a typo or is there such a thing somewhere? [...]

...

Also, when reading code, it may be hard to tell which (instance) attributes the class implements. To have these listed in the beginning of the class could therefore improve the readability.

Right. That has been my observation using PEP 484's type comments extensively for annotating instance variables at the class level. E.g. much of mypy's own code is written this way, and it really is a huge help. But

foo = None # type: List[int]

while it gives me the info I'm looking for, is not great notation-wise, and that's why I started thinking about an alternative:

foo: List[int]

(in either case, the __init__ contains something like `self.foo = []`).

...

In this light, I'm not sure it's a good idea to allow attribute type hints inside methods.

Those are meant for the coding style where all attributes are initialized in the method and people just want to add annotations there. This is already in heavy use in some PEP-484-annotated code bases I know of, using # type comments, and I think it will be easier to get people to switch to syntactic annotations if they can mechanically translate those uses. (In fact we are planning an automatic translator.)

I suppose the translator would be somewhat more complicated if it were to move the type hints to the beginning of the class suite. Anyway, I hope there will at least be a recommendation somewhere (PEP 8?) to not mix the two styles of attribute annotation (beginning of class / in method). The whole readability benefit turns against itself if there are some non-ClassVar variables annotated outside __init__ and then the rest somewhere in __init__ and in whatever initialization helper methods __init__ happens to call. [...]

...

Note that we could then also have this:

def NAME

Which would, again for readability (see above), be a way to express that "there is an instance variable called X, but no type hint for now". I can't think of a *good* way to do this with the keyword-free version for people that don't use type hints.

And then there could also be a simple decorator like @slotted_attributes that automatically generates "__slots__" from the annotations.

This I like, or something like it. It can be a follow-up design. (I.e. a separate PEP, once we have experiece with PEP 526.)

I think there should be a syntax for this that does not involve type hints, but I can't seem to come up with anything that works with the keyword-free version :(. -- Koos

-- --Guido van Rossum (python.org/~guido)

-- + Koos Zevenhoven + http://twitter.com/k7hoven +

On Thu, Sep 01, 2016 at 04:11:05PM +0300, Koos Zevenhoven wrote:

Maybe it's just me, but I've always thought 'def' is Python's least logically used keyword. It seems to come from 'define', but what is it about 'define' that makes it relate to functions only.

Convention. You can't use "def" to define both functions and classes: def function(x): ... def Class(x): ... is ambiguous, which is the function and which is the class? So we cannot avoid at least one limitation: "def is for functions, or classes, but not both". Given that, it isn't that weird to make the rule "def is only for functions". [...]

Note that we could then also have this:

def NAME

Which would, again for readability (see above), be a way to express that "there is an instance variable called X, but no type hint for now". I can't think of a *good* way to do this with the keyword-free version for people that don't use type hints.

The simplest way would be to say "go on, one type hint won't hurt, there's no meaningful runtime cost, just do it". from typing import Any class X: NAME: Any Since I'm not running a type checker, it doesn't matter what hint I use, but Any is probably the least inaccurate. But I think there's a better way. Unless I've missed something, there's no way to pre-declare an instance attribute without specifying a type. (Even if that type is Any.) So how about we allow None as a type-hint on its own: NAME: None as equivalent to a declaration *without* a hint. The reader, and the type-checker, can see that there's an instance attribute called NAME, but in the absense of an actual hint, the type will have to be inferred, just as if it wasn't declared at all. The risk is that somebody will "helpfully" correct the "obvious typo" and change it to NAME = None, but I think that will usually be harmless. I can invent examples where they will behave differently, but they feel contrived to me: class X: spam: None # declaration only, without a hint def method(self): if not hasattr(self, "spam"): raise XError("spam not set") return self.spam def setup(self, arg): if hasattr(self, "spam"): raise XError("spam already set") self.spam = arg Changing the declaration to an assignment does change the behaviour of the class, but I think that will be obvious when it happens. -- Steve

On Thu, Sep 1, 2016 at 9:30 AM, Ivan Levkivskyi <levkivskyi@gmail.com> wrote:

On 1 September 2016 at 18:21, Steven D'Aprano <steve@pearwood.info> wrote: [...]

...

Unless I've missed something, there's no way to pre-declare an instance attribute without specifying a type. (Even if that type is Any.) So how about we allow None as a type-hint on its own:

NAME: None

as equivalent to a declaration *without* a hint. The reader, and the type-checker, can see that there's an instance attribute called NAME, but in the absense of an actual hint, the type will have to be inferred, just as if it wasn't declared at all.

There is a convention for function annotations in PEP 484 that a missing annotation is equivalent to Any, so that I like your first option more.

But Steven wasn't proposing it to mean Any, he was proposing it to mean "type checker should infer". Where I presume the inference should be done based on the assignment in __init__ only. I'm not sure if this needs special syntax (a type checker might behave the same way without this, so we could just use a comment) but even if we did decide we wanted to support NAME: None for this case, we don't have to change Python, since this already conforms to the syntax in PEP 526 (the type is None). We'd still have to update the PEP to tell the authors of type checkers about this special feature, since otherwise it would mean "NAME has type NoneType" (remember that PEP 484 defines None as a shortcut for NoneType == type(None)). But that's not a very useful type for a variable... But I'm not in a hurry for that -- I'm only hoping to get the basic syntax accepted by Python 3.6 beta 1 so that we can start using this in 5 years from now rather than 7 years from now. -- --Guido van Rossum (python.org/~guido)

On Fri, Sep 2, 2016 at 6:43 AM, Ivan Levkivskyi <levkivskyi@gmail.com> wrote:

On 2 September 2016 at 04:38, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

However, a standalone Ellipsis doesn't currently have a meaning as a type annotation (it's only meaningful when subscripting Tuple and Callable), so a spelling like this might work:

NAME: ...

That spelling could then also be used in function definitions to say "infer the return type from the return statements rather than assuming Any"

Interesting idea. This is somehow similar to one of the existing use of Ellipsis: in numpy it infers how many dimensions needs to have the full slice, it is like saying "You know what I mean". So I am +1 on this solution.

I like it too, but I think it's better to leave any firm promises about the *semantics* of variable annotations out of the PEP. I just spoke to someone who noted that the PEP is likely to evoke an outsize emotional response. (Similar to what happened with PEP 484.) Pinning down the semantics is not why I am pushing for PEP 526 -- I only want to pin down the *syntax* to the point where we won't have to change it again for many versions, since it's much harder to change the syntax than it is to change the behavior of type checkers (which have fewer backwards compatibility constraints, a faster release cycle, and narrower user bases than core Python itself). -- --Guido van Rossum (python.org/~guido)

Guido van Rossum writes:

I just spoke to someone who noted that [PEP 526] is likely to evoke an outsize emotional response. (Similar to what happened with PEP 484.)

Emotional, yes, but I resent the "outsize" part. Although that word to the wise is undoubtedly enough, i.e., tl:dr if you like, let me explain why I have one foot in each camp. Compare Nick's version of "scientific code with SI units": from circuit_units import A, V, Ohm, seconds delta: A for delta in [-500n, 0, 500n]: input: A = 2.75u + delta wait(seconds(1u)) expected: V = Ohm(100k)*input tolerance: V = 2.2m fails = check_output(expected, tolerance) print('%s: I(in)=%rA, measured V(out)=%rV, expected V(out)=%rV, diff=%rV.' % ( 'FAIL' if fails else 'pass', input, get_output(), expected, get_output() - expected )) with from circuit_units import VoltType, uA, mV, kOhm, u_second expected: VoltType for delta in [-0.5*uA, 0*uA, 0.5*uA]: input = 2.75*uA + delta wait(1*u_second) expected = (100*kOhm)*input tolerance = 2.2*mV fails = check_output(expected, tolerance) print('%s: I(in)=%rA, measured V(out)=%rV, expected V(out)=%rV, diff=%rV.' % ( 'FAIL' if fails else 'pass', input, get_output(), expected, get_output() - expected )) In Nick's version, literals like 500n ("500 nano-whatevers") require Ken Kundert's proposed syntax change. I left that in because it streamlines the expressions. I wrote the latter because I really disliked Nick's version, streamlined with "SI scale syntax" or not. Nick didn't explicitly type the *_output functions, so I didn't either.[1] I assume they're annotated in their module of definition. The important point about the second version is that if we accept the hypothesis that the pseudo-literals like '[0.5*uA, 0*uA, 0.5*uA]' are good enough to implicitly type the variables they're assigned to (as they are in this snippet), mypy will catch "unit errors" (which the circuit_units module converts into TypeErrors) in the expressions. I think that this hypothesis is appropriate in the context of the thread. Therefore, I think Nick's version was an abuse of variable annotation. I don't mean to criticize Nick, as he was trying to make the best of an unlikely proposal. But if Nick can fall into this trap[2], I think the fears of many that type annotations will grow like fungus on code that really doesn't need them, and arguably is better without them, are quite reasonable. The point here is *not* that Nick's version is "horrible" (as many of the "emotional" type refuseniks might say), whatever that might mean. I can easily imagine that the snippet above is part of the embedded software for air traffic control or medical life support equipment, and a belt and suspenders approach ("make units visible to reviewers" + "mypy checking" + "full coverage in unit tests") is warranted. Ie, Nick's version is much better than mine in that context because the hypothesis that "implicit declaration" is good enough is invalid. But in the context of discussion of how to make measurement units visible and readable in a Python program, he grabbed an inappropriate tool because it was close to hand. My version does everything the OP asked for, it furthermore makes mypy into a units checker, and it does so in a way that any intermediate Python programmer (and many novices as well) can immediately grasp. If Nick had been given the constraint that novices should be able to read it, I suspect he would have written the same snippet I did. Nick? Footnotes: [1] I think both versions could have better variable naming along with a few other changes to make them more readable, but those aspects were dictated by an earlier post. I don't have the nerve to touch Nick's code, so left those as is. [2] Or perhaps he did it intentionally, trying to combine two cool ideas, variable type annotations and SI units, in one example. If so, I think that this combination was unwise in context.

Stephen J. Turnbull writes:

My version ... furthermore makes mypy into a units checker,

That isn't true, mypy does want annotations on all the variables it checks and does not infer them from initializer type. Sorry for the misinformation.

On Sat, Sep 3, 2016 at 8:18 AM, Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:

Stephen J. Turnbull writes:

...

My version ... furthermore makes mypy into a units checker,

That isn't true, mypy does want annotations on all the variables it checks and does not infer them from initializer type.

But it does! Mypy emphatically does *not* need annotations on all variables; it infers most variable types from the first expression assigned to them. E.g. here: output = [] n = 0 output.append(n) reveal_type(output) it will reveal the type List[int] without any help from annotations. There are cases where it does require annotation on empty containers, when it's less obvious how the container is filled, and other, more complicated situations, but a sequence of assignments as in Nick's example is a piece of cake for it. In fact, the one place where *I* wanted a type annotation was here: expected: V = Ohm(100k)*input because I haven't had a need to use Ohm's law in a long time, so I could personally use the hint that Ohm times Amps makes Volts (but again, given suitable class definitions, mypy wouldn't have needed that annotation). -- --Guido van Rossum (python.org/~guido)

On 2 September 2016 at 02:21, Steven D'Aprano <steve@pearwood.info> wrote:

Unless I've missed something, there's no way to pre-declare an instance attribute without specifying a type. (Even if that type is Any.) So how about we allow None as a type-hint on its own:

NAME: None

None already has a meaning as an annotation - it's a shorthand for "type(None)". While for variables and parameters, that's usually only seen in combination with Union, and even though Union[T, None] has a preferred spelling as Optional[T], there's also the "-> None" case to specify that a function doesn't return a value. Having "-> None" mean "no return value" and "NAME: None" mean "infer type from later assignment" would be quite confusing. However, a standalone Ellipsis doesn't currently have a meaning as a type annotation (it's only meaningful when subscripting Tuple and Callable), so a spelling like this might work: NAME: ... That spelling could then also be used in function definitions to say "infer the return type from the return statements rather than assuming Any": def inferred_return_type(): -> ... return some_other_function() Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, Sep 1, 2016 at 10:30 AM, Steven D'Aprano <steve@pearwood.info> wrote:

On Tue, Aug 30, 2016 at 02:20:26PM -0700, Guido van Rossum wrote:

...

- Whether (given PEP 484's relative success) it's worth adding syntax for variable/attribute annotations.

The PEP makes a good case that it does.

Thanks, I agree. :-)

...

- Whether the keyword-free syntax idea proposed here is best: NAME: TYPE TARGET: TYPE = VALUE

I think so.

That looks like similar to the syntax used by TypeScript:

http://www.typescriptlang.org/docs/handbook/type-inference.html

let zoo: Animal[] = [new Rhino(), new Elephant(), new Snake()];

And Rust. In the tracker issue we're still tweaking this, e.g. the latest idea is that after all we'd like to simplify the syntax to TARGET: TYPE [= VALUE] Please read the end of the tracker discussion: https://github.com/python/typing/issues/258#issuecomment-244188268

Some additional thoughts:

Is it okay to declare something as both an instance and class attribute?

class X: spam: int spam: ClassVar[Str] = 'suprise!'

def __init__(self): self.spam = 999

I would expect it should be okay.

I think it would be confusing because the class var would be used as the default if the instance var is not defined.

It is more common in Python circles to talk about class and instance *attributes* than "variables". Class variable might be okay in a language like Java where classes themselves aren't first-class values, but in Python "class variable" always makes me think it is talking about a variable which is a class, just like a string variable or list variable. Can we have ClassAttr[] instead of ClassVar[]?

We went back and forth on this. I really don't like to use the word attribute here, because a method is also an attribute. And instance variable sounds more natural to me than instance attribute. Also we now have global variables, class variables, instance variables, and local variables, all of which can be annotated. (The PEP's language is actually a bit inconsistent here.)

Other than that, +1 on the PEP.

-- --Guido van Rossum (python.org/~guido)