New subject: Conventions for Python code documentation (was: PEP 484 (Type Hints) -- first draft round)

16 Jan 2015

      After a long editing process we've got PEP 484
https://www.python.org/dev/peps/pep-0484/ (Type Hints) ready for your
review. This is by no means final, and several areas are either open (to be
resolved in a later draft) or postponed (to a different PEP altogether).
But there's enough meat that I think we can start having the discussion.
Please also see PEP 483 https://www.python.org/dev/peps/pep-0483/ (The
Theory of Type Hint; copied and reformatted from the original Quip document
that I posted just before last Christmas) and PEP 482
https://www.python.org/dev/peps/pep-0482/ (Literature Overview for Type
Hints, by Łukasz). Those are informational PEPs though; the actual spec is
focused in PEP 484 (the only one on the Standards Track).

As I said earlier, I hope to have a rough consensus before PyCon
https://us.pycon.org/2015/ and working code (just the typing.py module
https://github.com/ambv/typehinting/tree/master/prototyping, really)
committed to CPython before the last 3.5 alpha
https://www.python.org/dev/peps/pep-0478/.

Here is the raw text of PEP 484. Fire away!!

PEP: 484
Title: Type Hints
Version: $Revision$
Last-Modified: $Date$
Author: Guido van Rossum , Jukka Lehtosalo <
jukka.lehtosalo@iki.fi>, Łukasz Langa 
Discussions-To: Python-Dev 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 29-Sep-2014
Post-History: 16-Jan-2015
Resolution:

Abstract
========

This PEP introduces a standard syntax for type hints using annotations
on function definitions.

The proposal is strongly inspired by mypy [mypy]_.

The theory behind type hints and gradual typing is explained in PEP 483.

Rationale and Goals
===================

PEP 3107 added support for arbitrary annotations on parts of a function
definition.  Although no meaning was assigned to annotations then, there
has always been an implicit goal to use them for type hinting, which is
listed as the first possible use case in said PEP.

This PEP aims to provide a standard syntax for type annotations, opening
up Python code to easier static analysis and refactoring, potential
runtime type checking, and performance optimizations utilizing type
information.

Type Definition Syntax
======================

The syntax leverages PEP 3107-style annotations with a number of
extensions described in sections below.  In its basic form, type hinting
is used by filling function annotations with classes::

  def greeting(name: str) -> str:
      return 'Hello ' + name

This denotes that the expected type of the ``name`` argument is ``str``.
Analogically, the expected return type is ``str``.  Subclasses of
a specified argument type are also accepted as valid types for that
argument.

Abstract base classes, types available in the ``types`` module, and
user-defined classes may be used as type hints as well.  Annotations
must be valid expressions that evaluate without raising exceptions at
the time the function is defined.  In addition, the needs of static
analysis require that annotations must be simple enough to be
interpreted by static analysis tools.  (This is an intentionally
somewhat vague requirement.)

.. FIXME: Define rigorously what is/isn't supported.

When used as an annotation, the expression ``None`` is considered
equivalent to ``NoneType`` (i.e., ``type(None)`` for type hinting
purposes.

Type aliases are also valid type hints::

  integer = int

  def retry(url: str, retry_count: integer): ...

New names that are added to support features described in following
sections are available in the ``typing`` package.

Callbacks
---------

Frameworks expecting callback functions of specific signatures might be
type hinted using ``Callable[[Arg1Type, Arg2Type], ReturnType]``.
Examples::

  from typing import Any, AnyArgs, Callable

  def feeder(get_next_item: Callable[[], Item]): ...

  def async_query(on_success: Callable[[int], None], on_error:
Callable[[int, Exception], None]): ...

  def partial(func: Callable[AnyArgs, Any], *args): ...

Since using callbacks with keyword arguments is not perceived as
a common use case, there is currently no support for specifying keyword
arguments with ``Callable``.

Generics
--------

Since type information about objects kept in containers cannot be
statically inferred in a generic way, abstract base classes have been
extended to support subscription to denote expected types for container
elements.  Example::

  from typing import Mapping, Set

  def notify_by_email(employees: Set[Employee], overrides: Mapping[str,
str]): ...

Generics can be parametrized by using a new factory available in
``typing`` called ``TypeVar``.  Example::

  from typing import Sequence, TypeVar

  T = TypeVar('T')      # Declare type variable

  def first(l: Sequence[T]) -> T:   # Generic function
      return l[0]

In this case the contract is that the returning value is consistent with
the elements held by the collection.

``TypeVar`` supports constraining parametric types to classes with any of
the specified bases.  Example::

  from typing import Iterable

  X = TypeVar('X')
  Y = TypeVar('Y', Iterable[X])

  def filter(rule: Callable[[X], bool], input: Y) -> Y:
      ...

.. FIXME: Add an example with multiple bases defined.

In the example above we specify that ``Y`` can be any subclass of
Iterable with elements of type ``X``, as long as the return type of
``filter()`` will be the same as the type of the ``input``
argument.

.. FIXME: Explain more about how this works.

Forward references
------------------

When a type hint contains names that have not been defined yet, that
definition may be expressed as a string, to be resolved later.  For
example, instead of writing::

  def notify_by_email(employees: Set[Employee]): ...

one might write::

  def notify_by_email(employees: 'Set[Employee]'): ...

.. FIXME: Rigorously define this.  Defend it, or find an alternative.

Union types
-----------

Since accepting a small, limited set of expected types for a single
argument is common, there is a new special factory called ``Union``.
Example::

  from typing import Union

  def handle_employees(e: Union[Employee, Sequence[Employee]]):
      if isinstance(e, Employee):
          e = [e]
      ...

A type factored by ``Union[T1, T2, ...]`` responds ``True`` to
``issubclass`` checks for ``T1`` and any of its subclasses, ``T2`` and
any of its subclasses, and so on.

One common case of union types are *optional* types.  By default,
``None`` is an invalid value for any type, unless a default value of
``None`` has been provided in the function definition.  Examples::

  def handle_employee(e: Union[Employee, None]): ...

As a shorthand for ``Union[T1, None]`` you can write ``Optional[T1]``;
for example, the above is equivalent to::

  from typing import Optional

  def handle_employee(e: Optional[Employee]): ...

An optional type is also automatically assumed when the default value is
``None``, for example::

  def handle_employee(e: Employee = None): ...

This is equivalent to::

  def handle_employee(e: Optional[Employee] = None): ...

.. FIXME: Is this really a good idea?

A special kind of union type is ``Any``, a class that responds
``True`` to ``issubclass`` of any class.  This lets the user
explicitly state that there are no constraints on the type of a
specific argument or return value.

Platform-specific type checking
-------------------------------

In some cases the typing information will depend on the platform that
the program is being executed on.  To enable specifying those
differences, simple conditionals can be used::

  from typing import PY2, WINDOWS

  if PY2:
      text = unicode
  else:
      text = str

  def f() -> text: ...

  if WINDOWS:
      loop = ProactorEventLoop
  else:
      loop = UnixSelectorEventLoop

Arbitrary literals defined in the form of ``NAME = True`` will also be
accepted by the type checker to differentiate type resolution::

  DEBUG = False
  ...
  if DEBUG:
      class Tracer:
          <verbose implementation>
  else:
      class Tracer:
          <dummy implementation>

For the purposes of type hinting, the type checker assumes ``__debug__``
is set to ``True``, in other words the ``-O`` command-line option is not
used while type checking.

Compatibility with other uses of function annotations
-----------------------------------------------------

A number of existing or potential use cases for function annotations
exist, which are incompatible with type hinting.  These may confuse a
static type checker.  However, since type hinting annotations have no
run time behavior (other than evaluation of the annotation expression
and storing annotations in the ``__annotations__`` attribute of the
function object), this does not make the program incorrect -- it just
makes it issue warnings when a static analyzer is used.

To mark portions of the program that should not be covered by type
hinting, use the following:

* a ``@no_type_checks`` decorator on classes and functions

* a ``# type: ignore`` comment on arbitrary lines

.. FIXME: should we have a module-wide comment as well?

Type Hints on Local and Global Variables
========================================

No first-class syntax support for explicitly marking variables as being
of a specific type is added by this PEP.  To help with type inference in
complex cases, a comment of the following format may be used::

  x = []   # type: List[Employee]

In the case where type information for a local variable is needed before
if was declared, an ``Undefined`` placeholder might be used::

  from typing import Undefined

  x = Undefined   # type: List[Employee]
  y = Undefined(int)

If type hinting proves useful in general, a syntax for typing variables
may be provided in a future Python version.

Explicit raised exceptions
==========================

No support for listing explicitly raised exceptions is being defined by
this PEP.  Currently the only known use case for this feature is
documentational, in which case the recommendation is to put this
information in a docstring.

The ``typing`` package
======================

To open the usage of static type checking to Python 3.5 as well as older
versions, a uniform namespace is required.  For this purpose, a new
package in the standard library is introduced called ``typing``.  It
holds a set of classes representing builtin types with generics, namely:

* Dict, used as ``Dict[key_type, value_type]``

* List, used as ``List[element_type]``

* Set, used as ``Set[element_type]``. See remark for ``AbstractSet``
  below.

* FrozenSet, used as ``FrozenSet[element_type]``

* Tuple, used as ``Tuple[index0_type, index1_type, ...]``.
  Arbitrary-length tuples might be expressed using ellipsis, in which
  case the following arguments are considered the same type as the last
  defined type on the tuple.

It also introduces factories and helper members needed to express
generics and union types:

* Any, used as ``def get(key: str) -> Any: ...``

* Union, used as ``Union[Type1, Type2, Type3]``

* TypeVar, used as ``X = TypeVar('X', Type1, Type2, Type3)`` or simply
  ``Y = TypeVar('Y')``

* Undefined, used as ``local_variable = Undefined # type: List[int]`` or
  ``local_variable = Undefined(List[int])`` (the latter being slower
  during runtime)

* Callable, used as ``Callable[[Arg1Type, Arg2Type], ReturnType]``

* AnyArgs, used as ``Callable[AnyArgs, ReturnType]``

* AnyStr, equivalent to ``TypeVar('AnyStr', str, bytes)``

All abstract base classes available in ``collections.abc`` are
importable from the ``typing`` package, with added generics support:

* ByteString

* Callable

* Container

* Hashable

* ItemsView

* Iterable

* Iterator

* KeysView

* Mapping

* MappingView

* MutableMapping

* MutableSequence

* MutableSet

* Sequence

* Set as ``AbstractSet``. This name change was required because ``Set``
  in the ``typing`` module means ``set()`` with generics.

* Sized

* ValuesView

* Mapping

The library includes literals for platform-specific type hinting:

* PY2

* PY3, equivalent to ``not PY2``

* WINDOWS

* UNIXOID, equivalent to ``not WINDOWS``

The following types are available in the ``typing.io`` module:

* IO

* BinaryIO

* TextIO

The following types are provided by the ``typing.re`` module:

* Match and Pattern, types of ``re.match()`` and ``re.compile()``
  results

As a convenience measure, types from ``typing.io`` and ``typing.re`` are
also available in ``typing`` (quoting Guido, "There's a reason those
modules have two-letter names.").

The place of the ``typing`` module in the standard library
----------------------------------------------------------

.. FIXME: complete this section

Usage Patterns
==============

The main use case of type hinting is static analysis using an external
tool without executing the analyzed program.  Existing tools used for
that purpose like ``pyflakes`` [pyflakes]_ or ``pylint`` [pylint]_
might be extended to support type checking.  New tools, like mypy's
``mypy -S`` mode, can be adopted specifically for this purpose.

Type checking based on type hints is understood as a best-effort
mechanism.  In other words, whenever types are not annotated and cannot
be inferred, the type checker considers such code valid.  Type errors
are only reported in case of explicit or inferred conflict.  Moreover,
as a mechanism that is not tied to execution of the code, it does not
affect runtime behaviour.  In other words, even in the case of a typing
error, the program will continue running.

The implementation of a type checker, whether linting source files or
enforcing type information during runtime, is out of scope for this PEP.

.. FIXME: Describe stub modules.

.. FIXME: Describe run-time behavior of generic types.

Existing Approaches
===================

PEP 482 lists existing approaches in Python and other languages.

Is type hinting Pythonic?
=========================

Type annotations provide important documentation for how a unit of code
should be used.  Programmers should therefore provide type hints on
public APIs, namely argument and return types on functions and methods
considered public.  However, because types of local and global variables
can be often inferred, they are rarely necessary.

The kind of information that type hints hold has always been possible to
achieve by means of docstrings.  In fact, a number of formalized
mini-languages for describing accepted arguments have evolved.  Moving
this information to the function declaration makes it more visible and
easier to access both at runtime and by static analysis.  Adding to that
the notion that “explicit is better than implicit”, type hints are
indeed *Pythonic*.

Acknowledgements
================

This document could not be completed without valuable input,
encouragement and advice from Jim Baker, Jeremy Siek, Michael Matson
Vitousek, Andrey Vlasovskikh, and Radomir Dopieralski.

Influences include existing languages, libraries and frameworks
mentioned in PEP 482.  Many thanks to their creators, in alphabetical
order: Stefan Behnel, William Edwards, Greg Ewing, Larry Hastings,
Anders Hejlsberg, Alok Menghrajani, Travis E. Oliphant, Joe Pamer,
Raoul-Gabriel Urma, and Julien Verlaguet.

References
==========

.. [mypy]
   http://mypy-lang.org

.. [pyflakes]
   https://github.com/pyflakes/pyflakes/

.. [pylint]
   http://www.pylint.org

Copyright
=========

This document has been placed in the public domain.

..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:

-- 
--Guido van Rossum (python.org/~guido)

PEP 484 (Type Hints) -- first draft round

Eugene Toder

Cem Karan

Cem Karan

Cem Karan

Cem Karan

Cem Karan

Cem Karan

Cem Karan

Cem Karan

Cem Karan

Eugene Toder