[Python-Dev] PEP 484 (Type Hints) -- the next draft is here
Guido van Rossum
guido at python.org
Fri Apr 17 23:58:25 CEST 2015
The PEP 484 authors (Jukka, Łukasz and I) have a new draft ready. This time
we're posting to python-dev.
While we don't have all details totally sorted out, it's a lot closer. We
have a BDFL-Delegate (Mark Shannon), and we've worked out a variety of
issues at PyCon. Our hope remains that we'll get the typing.py module added
to CPython 3.5 before beta 1 goes out (May 24).
Below is a list of key changes since the draft posted to python-dev on
March 20 (for more details see
https://github.com/ambv/typehinting/commits/master/pep-0484.txt) and the
full text of the PEP. The new draft is also in the peps repo (
https://hg.python.org/peps/file/tip/pep-0484.txt) and will soon be on
python.org (https://www.python.org/dev/peps/pep-0484/ -- give it 10-30
minutes).
As always, between draft postings the PEP text is updated frequently by the
authors in a dedicated GitHub repo (https://github.com/ambv/typehinting),
and many detailed discussions are found in the issue tracker there (
https://github.com/ambv/typehinting/issues). The typing.py module also
lives in that repo (
https://github.com/ambv/typehinting/tree/master/prototyping).
Key changes since 20-Mar-2015 draft
--------------------------------------------------
Generics:
- Properly specify generics and type variables.
- Add covariant/contravariant flags to TypeVar().
Type specifications:
- Define type aliases.
- Kill Undefined.
- Better specification of `# type:` comments.
Specifics:
- Add Generator (a generic type describing generator objects)
- Document get_type_hints().
Stubs:
- Only .pyi is a valid extension for stub files.
- Add discussion of third-party stubs (see issue #84).
Process:
- Mark Shannon is now BDFL-Delegate.
- Add section on PEP Development Process, with github links.
Other:
- Mention the __future__ proposal (a separate PEP to be developed).
- Lots of editing; remove some sections that didn't specify anything.
Full PEP text
------------------
PEP: 484
Title: Type Hints
Version: $Revision$
Last-Modified: $Date$
Author: Guido van Rossum <guido at python.org>, Jukka Lehtosalo <
jukka.lehtosalo at iki.fi>, Łukasz Langa <lukasz at langa.pl>
BDFL-delegate: Mark Shannon
Discussions-To: Python-Dev <python-dev at python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 29-Sep-2014
Post-History: 16-Jan-2015,20-Mar-2015,17-Apr-2015
Resolution:
Abstract
========
This PEP introduces a standard syntax for type hints using annotations
(PEP 3107) on function definitions. For example, here is a simple
function whose argument and return type are declared in the
annotations::
def greeting(name: str) -> str:
return 'Hello ' + name
While these annotations are available at runtime through the usual
``__annotations__`` attribute, *no type checking happens at runtime*.
Instead, the proposal assumes the existence of a separate off-line
type checker which users can run over their source code voluntarily.
Essentially, such a type checker acts as a very powerful linter.
The proposal is strongly inspired by mypy [mypy]_. For example, the
type "sequence of integers" can be written as ``Sequence[int]``. The
square brackets mean that no new syntax needs to be added to the
language. The example here uses a custom class ``Sequence``, imported
from a pure-Python module ``typing``. The ``Sequence[int]``
notation works by implementing ``__getitem__()`` in the metaclass.
The type system supports unions, generic types, and a special type
named ``Any`` which is consistent with (i.e. assignable to and from) all
types. This latter feature is taken from the idea of gradual typing.
Gradual typing and the full type system are explained in PEP 483.
Other approaches from which we have borrowed or to which ours can be
compared and contrasted are described in PEP 482.
Rationale and Goals
===================
PEP 3107 added support for arbitrary annotations on parts of a function
definition. Although no meaning was assigned to annotations then, there
has always been an implicit goal to use them for type hinting, which is
listed as the first possible use case in said PEP.
This PEP aims to provide a standard syntax for type annotations, opening
up Python code to easier static analysis and refactoring, potential
runtime type checking, and performance optimizations utilizing type
information.
Of these goals, static analysis is the most important. This includes
support for off-line type checkers such as mypy, as well as providing
a standard notation that can be used by IDEs for code completion and
refactoring.
Non-goals
---------
While the proposed typing module will contain some building blocks for
runtime type checking -- in particular a useful ``isinstance()``
implementation -- third party packages would have to be developed to
implement specific runtime type checking functionality, for example
using decorators or metaclasses. Using type hints for performance
optimizations is left as an exercise for the reader.
It should also be emphasized that Python will remain a dynamically
typed language, and the authors have no desire to ever make type hints
mandatory, even by convention.
What is checked?
================
Any function (or method -- for brevity we won't be repeating this)
with at least one argument or return annotation is checked, unless
type checking is disabled by the ``@no_type_check`` decorator or a
``# type: ignore`` comment (see below).
A checked function should have annotations for all its arguments and
its return type, with the exception that the ``self`` argument of a
method should not be annotated; it is assumed to have the type of the
containing class. Notably, the return type of ``__init__`` should be
annotated with ``-> None``.
The body of a checked function is checked for consistency with the
given annotations. The annotations are also used to check correctness
of calls appearing in other checked functions.
Functions without any annotations (or whose checking is disabled) are
assumed to have type ``Any`` when they are referenced in checked
functions, and this should completely silence complaints from the
checker regarding those references (although a checker may still
request that a type be specified using a cast or a ``# type:`` comment
if a more specific type than ``Any`` is needed for analysis of
subsequent code).
A type checker should understand decorators; this may require
annotations on decorator definitions. In particular, a type checker
should understand the built-in decorators ``@property``,
``@staticmethod`` and ``@classmethod``. The first argument of a class
method should not be annotated; it is assumed to be a subclass of the
defining class.
Type Definition Syntax
======================
The syntax leverages PEP 3107-style annotations with a number of
extensions described in sections below. In its basic form, type
hinting is used by filling function annotation slots with classes::
def greeting(name: str) -> str:
return 'Hello ' + name
This states that the expected type of the ``name`` argument is
``str``. Analogically, the expected return type is ``str``.
Expressions whose type is a subtype of a specific argument type are
also accepted for that argument.
Acceptable type hints
---------------------
Type hints may be built-in classes (including those defined in
standard library or third-party extension modules), abstract base
classes, types available in the ``types`` module, and user-defined
classes (including those defined in the standard library or
third-party modules). Annotations for built-in classes (and other
classes at the discretion of the developer) may be placed in stub
files (see below).
Annotations must be valid expressions that evaluate without raising
exceptions at the time the function is defined (but see below for
forward references).
The needs of static analysis require that annotations must be simple
enough to be interpreted by static analysis tools. In particular,
dynamically computed types are not acceptable. (This is an
intentionally somewhat vague requirement, specific inclusions and
exclusions may be added to future versions of this PEP as warranted by
the discussion.)
In addition to the above, the following special constructs defined
below may be used: ``None``, ``Any``, ``Union``, ``Tuple``,
``Callable``, all ABCs and stand-ins for concrete classes exported
from ``typing`` (e.g. ``Sequence`` and ``Dict``), type variables, and
type aliases.
All newly introducedw names used to support features described in
following sections (such as ``Any`` and ``Union``) are available in
the ``typing`` module.
Using None
----------
When used in a type hint, the expression ``None`` is considered
equivalent to ``type(None)``.
Type aliases
------------
Type aliases are defined by simple variable assignments::
Url = str
def retry(url: Url, retry_count: int) -> None: ...
Note that we recommend capitalizing alias names, since they represent
user-defined types, which (like user-defined classes) are typically
spelled that way.
Type aliases may be as complex as type hints in annotations --
anything that is acceptable as a type hint is acceptable in a type
alias::
from typing import TypeVar, Iterable, Tuple
T = TypeVar('T', int, float, complex)
Vector = Iterable[Tuple[T, T]]
def inproduct(v: Vector) -> T:
return sum(x*y for x, y in v)
This is equivalent to::
from typing import TypeVar, Iterable, Tuple
T = TypeVar('T', int, float, complex)
def inproduct(v: Iterable[Tuple[T, T]]) -> T:
return sum(x*y for x, y in v)
Callable
--------
Frameworks expecting callback functions of specific signatures might be
type hinted using ``Callable[[Arg1Type, Arg2Type], ReturnType]``.
Examples::
from typing import Callable
def feeder(get_next_item: Callable[[], str]) -> None:
# Body
def async_query(on_success: Callable[[int], None],
on_error: Callable[[int, Exception], None]) -> None:
# Body
It is possible to declare the return type of a callable without
specifying the call signature by substituting a literal ellipsis
(three dots) for the list of arguments::
def partial(func: Callable[..., str], *args) -> Callable[..., str]:
# Body
Note that there are no square brackets around the ellipsis. The
arguments of the callback are completely unconstrained in this case
(and keyword arguments are acceptable).
Since using callbacks with keyword arguments is not perceived as a
common use case, there is currently no support for specifying keyword
arguments with ``Callable``. Similarly, there is no support for
specifying callback signatures with a variable number of argument of a
specific type.
Generics
--------
Since type information about objects kept in containers cannot be
statically inferred in a generic way, abstract base classes have been
extended to support subscription to denote expected types for container
elements. Example::
from typing import Mapping, Set
def notify_by_email(employees: Set[Employee], overrides: Mapping[str,
str]) -> None: ...
Generics can be parametrized by using a new factory available in
``typing`` called ``TypeVar``. Example::
from typing import Sequence, TypeVar
T = TypeVar('T') # Declare type variable
def first(l: Sequence[T]) -> T: # Generic function
return l[0]
In this case the contract is that the returned value is consistent with
the elements held by the collection.
``TypeVar`` supports constraining parametric types to a fixed set of
possible types. For example, we can define a type variable that ranges
over just ``str`` and ``bytes``. By default, a type variable ranges
over all possible types. Example of constraining a type variable::
from typing import TypeVar
AnyStr = TypeVar('AnyStr', str, bytes)
def concat(x: AnyStr, y: AnyStr) -> AnyStr:
return x + y
The function ``concat`` can be called with either two ``str`` arguments
or two ``bytes`` arguments, but not with a mix of ``str`` and ``bytes``
arguments.
Note that subtypes of types constrained by a type variable are treated
as their respective explicitly listed base types in the context of the
type variable. Consider this example::
class MyStr(str): ...
x = concat(MyStr('apple'), MyStr('pie'))
The call is valid but the type variable ``AnyStr`` will be set to
``str`` and not ``MyStr``. In effect, the inferred type of the return
value assigned to ``x`` will also be ``str``.
Additionally, ``Any`` is a valid value for every type variable.
Consider the following::
def count_truthy(elements: List[Any]) -> int:
return sum(1 for elem in elements if element)
This is equivalent to omitting the generic notation and just saying
``elements: List``.
User-defined generic types
--------------------------
You can include a ``Generic`` base class to define a user-defined class
as generic. Example::
from typing import TypeVar, Generic
T = TypeVar('T')
class LoggedVar(Generic[T]):
def __init__(self, value: T, name: str, logger: Logger) -> None:
self.name = name
self.logger = logger
self.value = value
def set(self, new: T) -> None:
self.log('Set ' + repr(self.value))
self.value = new
def get(self) -> T:
self.log('Get ' + repr(self.value))
return self.value
def log(self, message: str) -> None:
self.logger.info('{}: {}'.format(self.name message))
``Generic[T]`` as a base class defines that the class ``LoggedVar``
takes a single type parameter ``T``. This also makes ``T`` valid as
a type within the class body.
The ``Generic`` base class uses a metaclass that defines ``__getitem__``
so that ``LoggedVar[t]`` is valid as a type::
from typing import Iterable
def zero_all_vars(vars: Iterable[LoggedVar[int]]) -> None:
for var in vars:
var.set(0)
A generic type can have any number of type variables, and type variables
may be constrained. This is valid::
from typing import TypeVar, Generic
...
T = TypeVar('T')
S = TypeVar('S')
class Pair(Generic[T, S]):
...
Each type variable argument to ``Generic`` must be distinct. This is
thus invalid::
from typing import TypeVar, Generic
...
T = TypeVar('T')
class Pair(Generic[T, T]): # INVALID
...
You can use multiple inheritance with ``Generic``::
from typing import TypeVar, Generic, Sized
T = TypeVar('T')
class LinkedList(Sized, Generic[T]):
...
Arbitrary generic types as base classes
---------------------------------------
``Generic[T]`` is only valid as a base class -- it's not a proper type.
However, user-defined generic types such as ``LinkedList[T]`` from the
above example and built-in generic types and ABCs such as ``List[T]``
and ``Iterable[T]`` are valid both as types and as base classes. For
example, we can define a subclass of ``Dict`` that specializes type
arguments::
from typing import Dict, List, Optional
class Node:
...
class SymbolTable(Dict[str, List[Node]]):
def push(self, name: str, node: Node) -> None:
self.setdefault(name, []).append(node)
def pop(self, name: str) -> Node:
return self[name].pop()
def lookup(self, name: str) -> Optional[Node]:
nodes = self.get(name)
if nodes:
return nodes[-1]
return None
``SymbolTable`` is a subclass of ``dict`` and a subtype of ``Dict[str,
List[Node]]``.
If a generic base class has a type variable as a type argument, this
makes the defined class generic. For example, we can define a generic
``LinkedList`` class that is iterable and a container::
from typing import TypeVar, Iterable, Container
T = TypeVar('T')
class LinkedList(Iterable[T], Container[T]):
...
Now ``LinkedList[int]`` is a valid type. Note that we can use ``T``
multiple times in the base class list, as long as we don't use the
same type variable ``T`` multiple times within ``Generic[...]``.
Abstract generic types
----------------------
The metaclass used by ``Generic`` is a subclass of ``abc.ABCMeta``.
A generic class can be an ABC by including abstract methods
or properties, and generic classes can also have ABCs as base
classes without a metaclass conflict.
Forward references
------------------
When a type hint contains names that have not been defined yet, that
definition may be expressed as a string literal, to be resolved later.
A situation where this occurs commonly is the definition of a
container class, where the class being defined occurs in the signature
of some of the methods. For example, the following code (the start of
a simple binary tree implementation) does not work::
class Tree:
def __init__(self, left: Tree, right: Tree):
self.left = left
self.right = right
To address this, we write::
class Tree:
def __init__(self, left: 'Tree', right: 'Tree'):
self.left = left
self.right = right
The string literal should contain a valid Python expression (i.e.,
``compile(lit, '', 'expr')`` should be a valid code object) and it
should evaluate without errors once the module has been fully loaded.
The local and global namespace in which it is evaluated should be the
same namespaces in which default arguments to the same function would
be evaluated.
Moreover, the expression should be parseable as a valid type hint, i.e.,
it is constrained by the rules from the section `Acceptable type hints`_
above.
It is allowable to use string literals as *part* of a type hint, for
example::
class Tree:
...
def leaves(self) -> List['Tree']:
...
Union types
-----------
Since accepting a small, limited set of expected types for a single
argument is common, there is a new special factory called ``Union``.
Example::
from typing import Union
def handle_employees(e: Union[Employee, Sequence[Employee]]) -> None:
if isinstance(e, Employee):
e = [e]
...
A type factored by ``Union[T1, T2, ...]`` responds ``True`` to
``issubclass`` checks for ``T1`` and any of its subclasses, ``T2`` and
any of its subclasses, and so on.
One common case of union types are *optional* types. By default,
``None`` is an invalid value for any type, unless a default value of
``None`` has been provided in the function definition. Examples::
def handle_employee(e: Union[Employee, None]) -> None: ...
As a shorthand for ``Union[T1, None]`` you can write ``Optional[T1]``;
for example, the above is equivalent to::
from typing import Optional
def handle_employee(e: Optional[Employee]) -> None: ...
An optional type is also automatically assumed when the default value is
``None``, for example::
def handle_employee(e: Employee = None): ...
This is equivalent to::
def handle_employee(e: Optional[Employee] = None) -> None: ...
The ``Any`` type
----------------
A special kind of type is ``Any``. Every class is a subclass of
``Any``. This is also true for the builtin class ``object``.
However, to the static type checker these are completely different.
When the type of a value is ``object``, the type checker will reject
almost all operations on it, and assigning it to a variable (or using
it as a return value) of a more specialized type is a type error. On
the other hand, when a value has type ``Any``, the type checker will
allow all operations on it, and a value of type ``Any`` can be assigned
to a variable (or used as a return value) of a more constrained type.
Predefined constants
--------------------
Some predefined Boolean constants are defined in the ``typing``
module to enable platform-specific type definitions and such::
from typing import PY2, PY3, WINDOWS, POSIX
if PY2:
text = unicode
else:
text = str
def f() -> text: ...
if WINDOWS:
loop = ProactorEventLoop
else:
loop = UnixSelectorEventLoop
It is up to the type checker implementation to define their values, as
long as ``PY2 == not PY3`` and ``WINDOWS == not POSIX``. When the
program is being executed these always reflect the current platform,
and this is also the suggested default when the program is being
type-checked.
Compatibility with other uses of function annotations
=====================================================
A number of existing or potential use cases for function annotations
exist, which are incompatible with type hinting. These may confuse
a static type checker. However, since type hinting annotations have no
runtime behavior (other than evaluation of the annotation expression and
storing annotations in the ``__annotations__`` attribute of the function
object), this does not make the program incorrect -- it just may cause
a type checker to emit spurious warnings or errors.
To mark portions of the program that should not be covered by type
hinting, you can use one or more of the following:
* a ``# type: ignore`` comment;
* a ``@no_type_check`` decorator on a class or function;
* a custom class or function decorator marked with
``@no_type_check_decorator``.
For more details see later sections.
In order for maximal compatibility with offline type checking it may
eventually be a good idea to change interfaces that rely on annotations
to switch to a different mechanism, for example a decorator. In Python
3.5 there is no pressure to do this, however. See also the longer
discussion under `Rejected alternatives`_ below.
Type comments
=============
No first-class syntax support for explicitly marking variables as being
of a specific type is added by this PEP. To help with type inference in
complex cases, a comment of the following format may be used::
x = [] # type: List[Employee]
x, y, z = [], [], [] # type: List[int], List[int], List[str]
x, y, z = [], [], [] # type: (List[int], List[int], List[str])
x = [
1,
2,
] # type: List[int]
Type comments should be put on the last line of the statement that
contains the variable definition. They can also be placed on
``with`` statements and ``for`` statements, right after the colon.
Examples of type comments on ``with`` and ``for`` statements::
with frobnicate() as foo: # type: int
# Here foo is an int
...
for x, y in points: # type: float, float
# Here x and y are floats
...
The ``# type: ignore`` comment should be put on the line that the
error refers to::
import http.client
errors = {
'not_found': http.client.NOT_FOUND # type: ignore
}
A ``# type: ignore`` comment on a line by itself disables all type
checking for the rest of the file.
If type hinting proves useful in general, a syntax for typing variables
may be provided in a future Python version.
Casts
=====
Occasionally the type checker may need a different kind of hint: the
programmer may know that an expression is of a more constrained type
than the type checker infers. For example::
from typing import List, cast
def find_first_str(a: List[object]) -> str:
index = next(i for i, x in enumerate(a) if isinstance(x, str))
# We only get here if there's at least one string in a
return cast(str, a[index])
The type checker infers the type ``object`` for ``a[index]``, but we
know that (if the code gets to that point) it must be a string. The
``cast(t, x)`` call tells the type checker that we are confident that
the type of ``x`` is ``t``. At runtime a cast always returns the
expression unchanged -- it does not check the type, and it does not
convert or coerce the value.
Casts differ from type comments (see the previous section). When using
a type comment, the type checker should still verify that the inferred
type is consistent with the stated type. When using a cast, the type
checker should blindly believe the programmer. Also, casts can be used
in expressions, while type comments only apply to assignments.
Stub Files
==========
Stub files are files containing type hints that are only for use by
the type checker, not at runtime. There are several use cases for
stub files:
* Extension modules
* Third-party modules whose authors have not yet added type hints
* Standard library modules for which type hints have not yet been
written
* Modules that must be compatible with Python 2 and 3
* Modules that use annotations for other purposes
Stub files have the same syntax as regular Python modules. There is one
feature of the ``typing`` module that may only be used in stub files:
the ``@overload`` decorator described below.
The type checker should only check function signatures in stub files;
function bodies in stub files should just be a single ``pass``
statement.
The type checker should have a configurable search path for stub files.
If a stub file is found the type checker should not read the
corresponding "real" module.
While stub files are syntactically valid Python modules, they use the
``.pyi`` extension to make it possible to maintain stub files in the
same directory as the corresponding real module. This also reinforces
the notion that no runtime behavior should be expected of stub files.
Function overloading
--------------------
The ``@overload`` decorator allows describing functions that support
multiple different combinations of argument types. This pattern is
used frequently in builtin modules and types. For example, the
``__getitem__()`` method of the ``bytes`` type can be described as
follows::
from typing import overload
class bytes:
...
@overload
def __getitem__(self, i: int) -> int: pass
@overload
def __getitem__(self, s: slice) -> bytes: pass
This description is more precise than would be possible using unions
(which cannot express the relationship between the argument and return
types)::
from typing import Union
class bytes:
...
def __getitem__(self, a: Union[int, slice]) -> Union[int, bytes]: pass
Another example where ``@overload`` comes in handy is the type of the
builtin ``map()`` function, which takes a different number of
arguments depending on the type of the callable::
from typing import Callable, Iterable, Iterator, Tuple, TypeVar, overload
T1 = TypeVar('T1')
T2 = TypeVar('T2)
S = TypeVar('S')
@overload
def map(func: Callable[[T1], S], iter1: Iterable[T1]) -> Iterator[S]: pass
@overload
def map(func: Callable[[T1, T2], S],
iter1: Iterable[T1], iter2: Iterable[T2]) -> Iterator[S]: pass
# ... and we could add more items to support more than two iterables
Note that we could also easily add items to support ``map(None, ...)``::
@overload
def map(func: None, iter1: Iterable[T1]) -> Iterable[T1]: pass
@overload
def map(func: None,
iter1: Iterable[T1],
iter2: Iterable[T2]) -> Iterable[Tuple[T1, T2]]: pass
The ``@overload`` decorator may only be used in stub files. While it
would be possible to provide a multiple dispatch implementation using
this syntax, its implementation would require using
``sys._getframe()``, which is frowned upon. Also, designing and
implementing an efficient multiple dispatch mechanism is hard, which
is why previous attempts were abandoned in favor of
``functools.singledispatch()``. (See PEP 443, especially its section
"Alternative approaches".) In the future we may come up with a
satisfactory multiple dispatch design, but we don't want such a design
to be constrained by the overloading syntax defined for type hints in
stub files.
Storing and distributing stub files
-----------------------------------
The easiest form of stub file storage and distribution is to put them
alongside Python modules in the same directory. This makes them easy to
find by both programmers and the tools. However, since package
maintainers are free not to add type hinting to their packages,
third-party stubs installable by ``pip`` from PyPI are also supported.
In this case we have to consider three issues: naming, versioning,
installation path.
This PEP does not provide a recommendation on a naming scheme that
should be used for third-party stub file packages. Discoverability will
hopefully be based on package popularity, like with Django packages for
example.
Third-party stubs have to be versioned using the lowest version of the
source package that is compatible. Example: FooPackage has versions
1.0, 1.1, 1.2, 1.3, 2.0, 2.1, 2.2. There are API changes in versions
1.1, 2.0 and 2.2. The stub file package maintainer is free to release
stubs for all versions but at least 1.0, 1.1, 2.0 and 2.2 are needed
to enable the end user type check all versions. This is because the
user knows that the closest *lower or equal* version of stubs is
compatible. In the provided example, for FooPackage 1.3 the user would
choose stubs version 1.1.
Note that if the user decides to use the "latest" available source
package, using the "latest" stub files should generally also work if
they're updated often.
Third-party stub packages can use any location for stub storage. The
type checker will search for them using PYTHONPATH. A default fallback
directory that is always checked is ``shared/typehints/python3.5/`` (or
3.6, etc.). Since there can only be one package installed for a given
Python version per environment, no additional versioning is performed
under that directory (just like bare directory installs by ``pip`` in
site-packages). Stub file package authors might use the following
snippet in ``setup.py``::
...
data_files=[
(
'shared/typehints/python{}.{}'.format(*sys.version_info[:2]),
pathlib.Path(SRC_PATH).glob('**/*.pyi'),
),
],
...
Exceptions
==========
No syntax for listing explicitly raised exceptions is proposed.
Currently the only known use case for this feature is documentational,
in which case the recommendation is to put this information in a
docstring.
The ``typing`` Module
=====================
To open the usage of static type checking to Python 3.5 as well as older
versions, a uniform namespace is required. For this purpose, a new
module in the standard library is introduced called ``typing``. It
holds a set of classes representing builtin types with generics, namely:
* Dict, used as ``Dict[key_type, value_type]``
* List, used as ``List[element_type]``
* Set, used as ``Set[element_type]``. See remark for ``AbstractSet``
below.
* FrozenSet, used as ``FrozenSet[element_type]``
* Tuple, used by listing the element types, for example
``Tuple[int, int, str]``.
Arbitrary-length homogeneous tuples can be expressed
using one type and ellipsis, for example ``Tuple[int, ...]``.
(The ``...`` here are part of the syntax.)
* NamedTuple, used as
``NamedTuple(type_name, [(field_name, field_type), ...])``
and equivalent to
``collections.namedtuple(type_name, [field_name, ...])``.
The generic versions of concrete collection types (``Dict``, ``List``,
``Set``, ``FrozenSet``, and homogeneous arbitrary-length ``Tuple``)
are mainly useful for annotating return values. For arguments, prefer
the abstract collection types defined below, e.g. ``Mapping``,
``Sequence`` or ``AbstractSet``.
The ``typing`` module defines the ``Generator`` type for return values
of generator functions. It is a subtype of ``Iterable`` and it has
additional type variables for the type accepted by the ``send()``
method and the return type of the generator:
* Generator, used as ``Generator[yield_type, send_type, return_type]``
It also introduces factories and helper members needed to express
generics and union types:
* Any, used as ``def get(key: str) -> Any: ...``
* Union, used as ``Union[Type1, Type2, Type3]``
* TypeVar, used as ``X = TypeVar('X', Type1, Type2, Type3)`` or simply
``Y = TypeVar('Y')``
* Callable, used as ``Callable[[Arg1Type, Arg2Type], ReturnType]``
* AnyStr, defined as ``TypeVar('AnyStr', str, bytes)``
All abstract base classes available in ``collections.abc`` are
importable from the ``typing`` module, with added generics support:
* ByteString
* Callable (see above)
* Container
* Hashable (not generic, but present for completeness)
* ItemsView
* Iterable
* Iterator
* KeysView
* Mapping
* MappingView
* MutableMapping
* MutableSequence
* MutableSet
* Sequence
* Set, renamed to ``AbstractSet``. This name change was required
because ``Set`` in the ``typing`` module means ``set()`` with
generics.
* Sized (not generic, but present for completeness)
* ValuesView
A few one-off types are defined that test for single special methods
(similar to ``Hashable`` or ``Sized``):
* Reversible, to test for ``__reversed__``
* SupportsAbs, to test for ``__abs__``
* SupportsFloat, to test for ``__float__``
* SupportsInt, to test for ``__int__``
* SupportsRound, to test for ``__round__``
The library includes literals for platform-specific type hinting:
* PY2
* PY3, equivalent to ``not PY2``
* WINDOWS
* POSIX, equivalent to ``not WINDOWS``
The following conveniece functions and decorators are exported:
* cast, described earlier
* no_type_check, a decorator to disable type checking per class or
function (see below)
* no_type_check_decorator, a decorator to create your own decorators
with the same meaning as ``@no_type_check`` (see below)
* overload, described earlier
* get_type_hints, a utility function to retrieve the type hints from a
function or method. Given a function or method object, it returns
a dict with the same format as ``__annotations__``, but evaluating
forward references (which are given as string literals) as expressions
in the context of the original function or method definition.
The following types are available in the ``typing.io`` module:
* IO (generic over ``AnyStr``)
* BinaryIO (a simple subclass of ``IO[bytes]``)
* TextIO (a simple subclass of ``IO[str]``)
The following types are provided by the ``typing.re`` module:
* Match and Pattern, types of ``re.match()`` and ``re.compile()``
results (generic over ``AnyStr``)
As a convenience measure, types from ``typing.io`` and ``typing.re`` are
also available in ``typing`` (quoting Guido, "There's a reason those
modules have two-letter names.").
Rejected Alternatives
=====================
During discussion of earlier drafts of this PEP, various objections
were raised and alternatives were proposed. We discuss some of these
here and explain why we reject them.
Several main objections were raised.
Which brackets for generic type parameters?
-------------------------------------------
Most people are familiar with the use of angular brackets
(e.g. ``List<int>``) in languages like C++, Java, C# and Swift to
express the parametrization of generic types. The problem with these
is that they are really hard to parse, especially for a simple-minded
parser like Python. In most languages the ambiguities are usually
dealy with by only allowing angular brackets in specific syntactic
positions, where general expressions aren't allowed. (And also by
using very powerful parsing techniques that can backtrack over an
arbitrary section of code.)
But in Python, we'd like type expressions to be (syntactically) the
same as other expressions, so that we can use e.g. variable assignment
to create type aliases. Consider this simple type expression::
List<int>
>From the Python parser's perspective, the expression begins with the
same four tokens (NAME, LESS, NAME, GREATER) as a chained comparison::
a < b > c # I.e., (a < b) and (b > c)
We can even make up an example that could be parsed both ways::
a < b > [ c ]
Assuming we had angular brackets in the language, this could be
interpreted as either of the following two::
(a<b>)[c] # I.e., (a<b>).__getitem__(c)
a < b > ([c]) # I.e., (a < b) and (b > [c])
It would surely be possible to come up with a rule to disambiguate
such cases, but to most users the rules would feel arbitrary and
complex. It would also require us to dramatically change the CPython
parser (and every other parser for Python). It should be noted that
Python's current parser is intentionally "dumb" -- a simple grammar is
easier for users to reason about.
For all these reasons, square brackets (e.g. ``List[int]``) are (and
have long been) the preferred syntax for generic type parameters.
They can be implemented by defining the ``__getitem__()`` method on
the metaclass, and no new syntax is required at all. This option
works in all recent versions of Python (starting with Python 2.2).
Python is not alone in this syntactic choice -- generic classes in
Scala also use square brackets.
What about existing uses of annotations?
----------------------------------------
One line of argument points out that PEP 3107 explicitly supports
the use of arbitrary expressions in function annotations. The new
proposal is then considered incompatible with the specification of PEP
3107.
Our response to this is that, first of all, the current proposal does
not introduce any direct incompatibilities, so programs using
annotations in Python 3.4 will still work correctly and without
prejudice in Python 3.5.
We do hope that type hints will eventually become the sole use for
annotations, but this will require additional discussion and a
deprecation period after the initial roll-out of the typing module
with Python 3.5. The current PEP will have provisional status (see
PEP 411) until Python 3.6 is released. The fastest conceivable scheme
would introduce silent deprecation of non-type-hint annotations in
3.6, full deprecation in 3.7, and declare type hints as the only
allowed use of annotations in Python 3.8. This should give authors of
packages that use annotations plenty of time to devise another
approach, even if type hints become an overnight success.
Another possible outcome would be that type hints will eventually
become the default meaning for annotations, but that there will always
remain an option to disable them. For this purpose the current
proposal defines a decorator ``@no_type_check`` which disables the
default interpretation of annotations as type hints in a given class
or function. It also defines a meta-decorator
``@no_type_check_decorator`` which can be used to decorate a decorator
(!), causing annotations in any function or class decorated with the
latter to be ignored by the type checker.
There are also ``# type: ignore`` comments, and static checkers should
support configuration options to disable type checking in selected
packages.
Despite all these options, proposals have been circulated to allow
type hints and other forms of annotations to coexist for individual
arguments. One proposal suggests that if an annotation for a given
argument is a dictionary literal, each key represents a different form
of annotation, and the key ``'type'`` would be use for type hints.
The problem with this idea and its variants is that the notation
becomes very "noisy" and hard to read. Also, in most cases where
existing libraries use annotations, there would be little need to
combine them with type hints. So the simpler approach of selectively
disabling type hints appears sufficient.
The problem of forward declarations
-----------------------------------
The current proposal is admittedly sub-optimal when type hints must
contain forward references. Python requires all names to be defined
by the time they are used. Apart from circular imports this is rarely
a problem: "use" here means "look up at runtime", and with most
"forward" references there is no problem in ensuring that a name is
defined before the function using it is called.
The problem with type hints is that annotations (per PEP 3107, and
similar to default values) are evaluated at the time a function is
defined, and thus any names used in an annotation must be already
defined when the function is being defined. A common scenario is a
class definition whose methods need to reference the class itself in
their annotations. (More general, it can also occur with mutually
recursive classes.) This is natural for container types, for
example::
class Node:
"""Binary tree node."""
def __init__(self, left: Node, right: None):
self.left = left
self.right = right
As written this will not work, because of the peculiarity in Python
that class names become defined once the entire body of the class has
been executed. Our solution, which isn't particularly elegant, but
gets the job done, is to allow using string literals in annotations.
Most of the time you won't have to use this though -- most *uses* of
type hints are expected to reference builtin types or types defined in
other modules.
A counterproposal would change the semantics of type hints so they
aren't evaluated at runtime at all (after all, type checking happens
off-line, so why would type hints need to be evaluated at runtime at
all). This of course would run afoul of backwards compatibility,
since the Python interpreter doesn't actually know whether a
particular annotation is meant to be a type hint or something else.
A compromise is possible where a ``__future__`` import could enable
turning *all* annotations in a given module into string literals, as
follows::
from __future__ import annotations
class ImSet:
def add(self, a: ImSet) -> List[ImSet]: ...
assert ImSet.add.__annotations__ == {'a': 'ImSet', 'return':
'List[ImSet]'}
Such a ``__future__`` import statement will be proposed in a separate
PEP.
The double colon
----------------
A few creative souls have tried to invent solutions for this problem.
For example, it was proposed to use a double colon (``::``) for type
hints, solving two problems at once: disambiguating between type hints
and other annotations, and changing the semantics to preclude runtime
evaluation. There are several things wrong with this idea, however.
* It's ugly. The single colon in Python has many uses, and all of
them look familiar because they resemble the use of the colon in
English text. This is a general rule of thumb by which Python
abides for most forms of punctuation; the exceptions are typically
well known from other programming languages. But this use of ``::``
is unheard of in English, and in other languages (e.g. C++) it is
used as a scoping operator, which is a very different beast. In
contrast, the single colon for type hints reads natural -- and no
wonder, since it was carefully designed for this purpose (the idea
long predates PEP 3107 [gvr-artima]_). It is also used in the same
fashion in other languages from Pascal to Swift.
* What would you do for return type annotations?
* It's actually a feature that type hints are evaluated at runtime.
* Making type hints available at runtime allows runtime type
checkers to be built on top of type hints.
* It catches mistakes even when the type checker is not run. Since
it is a separate program, users may choose not to run it (or even
install it), but might still want to use type hints as a concise
form of documentation. Broken type hints are no use even for
documentation.
* Because it's new syntax, using the double colon for type hints would
limit them to code that works with Python 3.5 only. By using
existing syntax, the current proposal can easily work for older
versions of Python 3. (And in fact mypy supports Python 3.2 and
newer.)
* If type hints become successful we may well decide to add new syntax
in the future to declare the type for variables, for example
``var age: int = 42``. If we were to use a double colon for
argument type hints, for consistency we'd have to use the same
convention for future syntax, perpetuating the ugliness.
Other forms of new syntax
-------------------------
A few other forms of alternative syntax have been proposed, e.g. the
introduction of a ``where`` keyword [roberge]_, and Cobra-inspired
``requires`` clauses. But these all share a problem with the double
colon: they won't work for earlier versions of Python 3. The same
would apply to a new ``__future__`` import.
Other backwards compatible conventions
--------------------------------------
The ideas put forward include:
* A decorator, e.g. ``@typehints(name=str, returns=str)``. This could
work, but it's pretty verbose (an extra line, and the argument names
must be repeated), and a far cry in elegance from the PEP 3107
notation.
* Stub files. We do want stub files, but they are primarily useful
for adding type hints to existing code that doesn't lend itself to
adding type hints, e.g. 3rd party packages, code that needs to
support both Python 2 and Python 3, and especially extension
modules. For most situations, having the annotations in line with
the function definitions makes them much more useful.
* Docstrings. There is an existing convention for docstrings, based
on the Sphinx notation (``:type arg1: description``). This is
pretty verbose (an extra line per parameter), and not very elegant.
We could also make up something new, but the annotation syntax is
hard to beat (because it was designed for this very purpose).
It's also been proposed to simply wait another release. But what
problem would that solve? It would just be procrastination.
PEP Development Process
=======================
A live draft for this PEP lives on GitHub [github]_. There is also an
issue tracker [issues]_, where much of the technical discussion takes
place.
The draft on GitHub is updated regularly in small increments. The
official PEPS repo [peps_] is (usually) only updated when a new draft
is posted to python-dev.
Acknowledgements
================
This document could not be completed without valuable input,
encouragement and advice from Jim Baker, Jeremy Siek, Michael Matson
Vitousek, Andrey Vlasovskikh, Radomir Dopieralski, Peter Ludemann,
and the BDFL-Delegate, Mark Shannon.
Influences include existing languages, libraries and frameworks
mentioned in PEP 482. Many thanks to their creators, in alphabetical
order: Stefan Behnel, William Edwards, Greg Ewing, Larry Hastings,
Anders Hejlsberg, Alok Menghrajani, Travis E. Oliphant, Joe Pamer,
Raoul-Gabriel Urma, and Julien Verlaguet.
References
==========
.. [mypy]
http://mypy-lang.org
.. [pyflakes]
https://github.com/pyflakes/pyflakes/
.. [pylint]
http://www.pylint.org
.. [gvr-artima]
http://www.artima.com/weblogs/viewpost.jsp?thread=85551
.. [roberge]
http://aroberge.blogspot.com/2015/01/type-hinting-in-python-focus-on.html
.. [github]
https://github.com/ambv/typehinting
.. [issues]
https://github.com/ambv/typehinting/issues
.. [peps]
https://hg.python.org/peps/file/tip/pep-0484.txt
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
--
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150417/00bbb75a/attachment-0001.html>
More information about the Python-Dev
mailing list