[Python-Dev] Please reject or postpone PEP 526

[Guido van Rossum]

On Fri, Sep 2, 2016 at 6:47 AM, Mark Shannon <mark at hotpy.org> wrote:
> Hi everyone,
>
> I think we should reject, or at least postpone PEP 526.
>
> PEP 526 represents a major change to the language, however there are, I
> believe, a number of technical flaws with the PEP.
>
> It is probable that with significant revisions it can be a worthwhile
> addition to the language, but that cannot happen in time for 3.6 beta 1
(in
> 11 days).
>
> PEP 526 claims to be an extension of PEP 484, but I don't think that is
> entirely correct.
> PEP 484 was primarily about defining the semantics of pre-existing syntax.
> PEP 526 is about adding new syntax.
> Historically the bar for adding new syntax has been set very high. I don't
> think that PEP 526, in its current form, reaches that bar.
>
> Below is a list of the issues I have with the PEP as it stands.
>
> In many cases it makes it more effort than type comments
> ========================================================
>
> Type hints should be as easy to use as possible, and that means pushing as
> much work as possible onto the checker, and not burdening the programmer.
>
> Attaching type hints to variables, rather than expressions, reduces the
> potential for inference. This makes it harder for programmer, but easier
for
> the checker, which is the wrong way around.
>
> For example,, given a function:
> def spam(x: Optional[List[int]])->None: ...
>
> With type comments, this is intuitively correct and should type check:
> def eggs(cond:bool):
> if cond:
> x = None
> else:
> x = [] # type: List[int]
> spam(x) # Here we can infer the type of x
>
> With PEP 526 we loose the ability to infer types.
> def eggs(cond:bool):
> if cond:
> x = None # Not legal due to type declaration below
> else:
> x: List[int] = []
> spam(x)
>
> So we need to use a more complex type
> def eggs(cond:bool):
> x: Optional[List[int]]
> if cond:
> x = None # Now legal
> else:
> x: = []
> spam(x)
>
> I don't think this improves readability.
> Whether this is an acceptable change is debatable, but it does need some
> debate.

It looks like you're misinterpreting the intent of the PEP. It does not
mean legislate the behavior of the type checker in this way. In mypy, the
first example is already rejected because it wants the annotation on the
first occurrence. The plan is for mypy not to change its behavior -- the
old form

TARGET = VALUE # type: TYPE

will be treated the same way as the new form

TARGET: TYPE = VALUE

(If you have a beef with what this means in mypy you should probably take
it up with mypy, not with PEP 526.)

> It limits the use of variables
> ==============================
>
> In Python a name (variable) is just a binding that refers to an object.
> A name only exists in a meaningful sense once an object has been assigned
to
> it. Any attempt to use that name, without an object bound to it, will
result
> in a NameError.

(Or UnboundLocalError, if the compiler knows there is an assignment to the
name anywhere in the same (function) scope.)

> PEP 526 makes variables more than just bindings, as any rebinding must
> conform to the given type. This looses us some of the dynamism for which
we
> all love Python.

Thanks for catching this; that's not the intent.

> Quoting from the PEP:
> ```
> a: int
> a: str # Static type checker will warn about this.
> ```
> In other words, it is illegal for a checker to split up the variable, even
> though it is straightforward to do so.

One of my co-authors has gone too far here. The intent is not to legislate
what should happen in this case but to leave it to the checker. In mypy,
the equivalent syntax using type comments is currently indeed rejected, but
we're considering a change here (https://github.com/python/mypy/issues/1174).
The PEP 526 syntax will not make a difference here.

> However, without the type declarations,
> ```
> a = 1
> a = "Hi"
> ```
> is just fine. Useless, but fine.

And believe me, I want to keep it this way. I will amend the example and
clarify the intent in the text.

> We should be free to add extra variables, whenever we choose, for clarity.
> For example,
> total = foo() - bar()
> should not be treated differently from:
> revenue = foo()
> tax = bar()
> total = revenue - tax
>
> If types are inferred, there is no problem.
> However, if they must be declared, then the use of meaningfully named
> variables is discouraged.

There is no mandate to declare variables! I actually see the main use of
variable annotations in class bodies where it can crate a lot of clarity
around which instance variables exist and what they mean.

> [A note about type-inference:
> Type inference is not a universal panacea, but it can make life a lot
easier
> for programmers in statically type languages.
> Languages like C# use local type inference extensively and it means that
> many variables often do not need their type declared. We should take care
> not to limit the ability of checkers to infer values and types and make
> programmers' lives easier.
> Within a function, type inference is near perfect, failing only
occasionally
> for some generic types.
> One place where type inference definitely breaks down is across calls,
which
> is why PEP 484 is necessary.
> ]

Totally agreed. But type annotations are not *just* for the checker. I am
regularly delighted when I find function annotations in code that I have to
read for the first time, because it helps my understanding. Many others at
Dropbox (where we have been doing a fairly large-scale experiment with the
introduction of mypy) agree.

> It is premature
> ===============
>
> There are still plenty of issues to iron out w.r.t. PEP 484 types. I don't
> think we should be adding more syntax, until we have a *precise* idea of
> what is required.
>
> PEP 484 states:
> "If type hinting proves useful in general, a syntax for typing variables
may
> be provided in a future Python version."
> Has it proved useful in general? I don't think it has. Maybe it will in
> future, but it hasn't yet.

PEP 526 does not alter this situation. It doesn't define new types, only
new places where types can be used syntactically, and it is careful to give
them the same syntactic description as PEP 484 (it's an expression).
Practical use of mypy has shown that we use `# type` comments on variables
with some regularity and not having them in the AST has been a problem for
tools. For example, we have had to teach flake8 and pylint about type
comments (so that we could continue to benefit from there "unused import"
and "undefined variable" tests), and in both cases it was a gross hack.

> It seems confused about class attributes and instance attributes
> ================================================================
>
> The PEP also includes a section of how to define class attributes and
> instance attributes. It seems that everything needs to be defined in the
> class scope, even it is not an attribute of the class, but of its
instances.
> This seems confusing, both to human reader and machine analyser.

And yet of course most other OO languages, like C++ and Java, also let you
define instance and class variables in the class body (and again with a
default of "instance", IIRC you have to use "static" in both languages for
class variables).

> Example from PEP 526:
>
> class Starship:
>
> captain: str = 'Picard'
> damage: int
> stats: ClassVar[Dict[str, int]] = {}
>
> def __init__(self, damage: int, captain: str = None):
> self.damage = damage
> if captain:
> self.captain = captain # Else keep the default
>
> With type hints as they currently exist, the same code is shorter and
> doesn't contaminate the class namespace with the 'damage' attribute.
>
> class Starship:
>
> captain = 'Picard'
> stats = {} # type: Dict[str, int]
>
> def __init__(self, damage: int, captain: str = None):
> self.damage = damage # Can infer type as int
> if captain:
> self.captain = captain # Can infer type as str

It is also harder for the human reader to discover that there's a `damage`
instance attribute. (Many classes I've reviewed at Dropbox have pages and
pages of __init__ code, defining dozens of instance variables, sometimes
using init-helper methods.)

For example, if you look at the code for asyncio's Future class you'll see
this block at the class level:
```
class Future:
"""..."""

# Class variables serving as defaults for instance variables.
_state = _PENDING
_result = None
_exception = None
_loop = None
_source_traceback = None

_blocking = False # proper use of future (yield vs yield from)

_log_traceback = False # Used for Python 3.4 and later
_tb_logger = None # Used for Python 3.3 only

def __init__(self, *, loop=None):
...

```
The terminology here is actually somewhat confused, but these are all
default values for instance variables. Because the defaults here are all
immutable, the assignments are put here instead of in __init__ to save a
little space in the dict and to make __init__ shorter (it only has to set
those instance variables that have mutable values).

There's also a highly technical reason for preferring that some instance
variables are given a default value in the class -- that way if an
exception happens in __init__ and there is recovery code that tries to use
some instance variable that __init__ hadn't initialized yet (e.g. in an
attempt to log the object) it avoids AttributeErrors for those variables
that have defaults set on the class. This has happened to me often enough
that it is now a standard idiom in my head.

> This isn't an argument against adding type syntax for attributes in
general,
> just that the form suggested in PEP 526 doesn't seem to follow Python
> semantics.

I'm happy to weaken the semantics as mandated in the PEP. In fact I had
thought it already doesn't mandate any semantics (apart from storing
certain forms of annotations in __annotations__, to match PEP 3107 and PEP
484), although I agree some examples have crept in that may appear more
normative than we meant them.

> One could imagine applying minimal PEP 526 style hints, with standard
Python
> semantics and relying on type inference, as follows:
>
> class Starship:
>
> captain = 'Picard'
> stats: Dict[str, int] = {}
>
> def __init__(self, damage: int, captain: str = None):
> self.damage = damage
> if captain:
> self.captain = captain
>
> The PEP overstates the existing use of static typing in Python
> ==============================================================
>
> Finally, in the rejected proposal section, under "Should we introduce
> variable annotations at all?" it states that "Variable annotations have
> already been around for almost two years in the form of type comments,
> sanctioned by PEP 484."
> I don't think that this is entirely true.
> PEP 484 was about the syntax for types, declaring parameter and return
> types, and declaring custom types to be generic.
> PEP 484 does include a description of type comments, but they are always
> annotations on assignment statements and were primarily intended for use
in
> stub files.

That is a mis-characterization of the intent of type comments in PEP 484;
they are not primarily meant for stubs (the only think I find tying the two
together is the use of "..." as the initial value in stubs).

> Please don't turn Python into some sort of inferior Java.
> There is potential in this PEP, but in its current form I think it should
be
> rejected.

Thanks for your feedback. We will be sure not to turn Python into Java! But
I am unconvinced that your objections are reason to reject the PEP -- you
seem to be fine with the general *syntax* proposed, your concerns are about
the specific rules to be used by a type checker. I expect we'll be arguing
about those for years to come -- maybe one day a PEP will come along that
ties the semantics of types down, but PEP 526 is not it.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160902/3e1c8427/attachment.html>