(For visibility, posted both to python-dev and Discourse.)
Over the last couple of months, ever since delaying PEP 563’s default
change in 3.10
https://mail.python.org/archives/list/python-dev@python.org/message/CLVXXPQ2...,
the Steering Council has been discussing and deliberating over PEP 563
(Postponed Evaluation of Annotations)
https://www.python.org/dev/peps/pep-0563/, PEP 649 (Deferred Evaluation
Of Annotations Using Descriptors)
https://www.python.org/dev/peps/pep-0649/, and type annotations in
general. We haven’t made a decision yet, but we want to give everyone an
update on where we’re at, what we’re thinking and why, see if there’s
consensus on any of this, and perhaps restart the discussion around the
options.
First off, as Barry already mentioned in a different thread
https://mail.python.org/archives/list/python-dev@python.org/message/4TY3MVJQ...,
the SC does not want to see type annotations as separate from the Python
language. We don’t think it would be good to have the syntax or the
semantics diverge, primarily because we don’t think users would see them as
separate. Any divergence would be hard to explain in documentation, hard to
reason about when reading code, and hard to delineate, to describe what is
allowed where. There’s a lot of nuance in this position (it doesn’t
necessarily mean that all valid syntax for typing uses has to have sensible
semantics for non-typing uses), but we’ll be working on something to
clarify that more, later.
We also believe that the runtime uses of type annotations, which PEP 563
didn’t really anticipate, are valid uses that Python should support. If
function annotations and type annotations had evolved differently, such as
being strings from the start, PEP 563 might have been sufficient. It’s
clear runtime uses of type annotations serve a real, sensible purpose, and
Python benefits from supporting them.
By and large, the SC views PEP 649 as a better way forward. If PEP 563 had
never come along, it would be a fairly easy decision to accept PEP 649. We
are still inclined to accept PEP 649. That would leave the consideration
about what to do with PEP 563 and existing from __future__ import
annotations directives. As far as we can tell, there are two reasons for
code to want to use PEP 563: being able to conveniently refer to names that
aren’t available until later in the code (i.e. forward references), and
reducing the overhead of type annotations. If PEP 649 satisfies all of the
objectives of PEP 563, is there a reason to keep supporting PEP 563’s
stringified annotations? Are there any practical, real uses of stringified
annotations that would not be served by PEP 649's deferred annotations?
If we no longer need to support PEP 563, can we simply make from __future__
import annotations enable PEP 649? We still may want a new future import
for PEP 649 as a transitory measure, but we could make PEP 563's future
import mean the same thing, without actually stringifying annotations. (The
reason to do this would be to reduce the combinatorial growth of the
support matrix, and to simplify the implementation of the parser.) This
would affect code that expects annotations to always be strings, but such
code would have to be poking directly at function objects (the
__annotations__ attribute), instead of using the advertised ways of getting
at annotations (like typing.get_type_hints()). This question in particular
is one in which the SC isn't yet of one mind.
Keeping the future import and stringified annotations around is certainly
an option, but we’re worried about the cost of the implementation, the
support cost, and the confusion for users (specifically, it is a future
import that will never become the future). If we do keep them, how long
would we keep them around? Should we warn about their use? If we warn about
the future import, is the noise and confusion this generates going to be
worth it? If we don't warn about them, how will we ever be able to turn
them off?
One thing we’re thinking of specifically for the future import, and for
other deprecations in Python, is to revisit the deprecation and warning
policy. We think it’s pretty clear that the policy we have right now
doesn’t exactly work. We used to have noisy DeprecationWarnings, which were
confusing to end users when they were not in direct control of the code. We
now have silent-by-default DeprecationWarnings, where the expectation is
that test frameworks surface these warnings. This avoids the problem of end
users being confused, but leaves the problem of the code’s dependencies
triggering the warning, and thus still warns users (developers) not
necessarily in a position to fix the problem, which in turn leads to them
silencing the warning and moving on. We need a better way to reach the
users in a position to update the code.
One idea is to rely on linters and IDEs to provide this signal, possibly
with a clear upgrade path for the code (e.g. a 2to3-like fixer for a
specific deprecation). Support for deprecations happened to be brought up on
the typing-sig mailing list not too long ago
https://mail.python.org/archives/list/typing-sig@python.org/thread/E24WTMQUT...,
as an addition to the pytype type checker and hopefully others (full
disclosure, Yilei is a team-mate of Thomas’s at Google).
This sounds like a reasonably user-friendly approach, but it would require
buy-in from linter/IDE developers, or an officially supported “Python
linter” project that we control. There’s also the question of support
timelines: most tooling supports a wider range of Python versions than just
the two years that we use in our deprecation policy. Perhaps we need to
revisit the policy, and consider deprecation timelines based on how many
Python versions library developers usually want to support.
The SC continues to discuss the following open questions, and we welcome
your input on them:
1.
Is it indeed safe to assume PEP 649 satisfies all reasonable uses of PEP
563? Are there cases of type annotations for static checking or runtime
use that PEP 563 enables, which would break with PEP 649?
2.
Is it safe to assume very little code would be poking directly at
__annotations__
attributes of function objects; effectively, to declare them
implementation details and let them not be strings even in code that
currently has the annotations future import?
3.
Is the performance of PEP 649 and PEP 563 similar enough that we can
outright discount it as a concern? Does anyone actually care about the
overhead of type annotations anymore? Are there other options to alleviate
this potential issue (like a process-wide switch to turn off annotations)?
4.
If we do not need to keep PEP 563 support, which would be a lot easier
on code maintenance and our support matrix, do we need to warn about the
semantics change? Can we silently accept (and ignore) the future import
once PEP 649 is in effect?
5.
If we do need a warning, how loud, and how long should it be around? At
the end of the deprecation period, should the future import be an error, or
simply be ignored?
6.
Are there other options we haven’t thought of for dealing with
deprecations like this one?
Like I said, the SC isn’t done deliberating on any of this. The only
decisions we’ve made so far is that we don’t see the typing language as
separate from Python (and thus won’t be blanket delegating typing PEPs to a
separate authority), and we don’t see type annotations as purely for static
analysis use.
For the whole SC,
Thomas.
--
Thomas Wouters