Mailman 3 March 2015 - Python-ideas

0-base and 1-base indexed iterables? Custom slicing rules?
by pedro santos March 24, 2015

March 24, 2015

Hi I'm an Art and CG student learning Python and today's exercise was about positions in a tiled room. The fact that I had to check if a position was inside the room and given that in a 1x1 room, 0.0 was considered in and 1.0 was considered out, it kept me thinking about 0-base indexing iterables and slicing. Read some articles and discussions, some pros and cons to each 0-base and 1-base, concerns about slicing, etc. But ultimately the question that got stuck in me and didn't found an answer was: Why can't both 0-base and 1-base indexing exist in the same language, and why can't slicing be customized? If I'm indexing the ruler marks, intervals, boundaries, dots, makes sense to start of at 0; rul=[0,1,2,3,4,5,6] would index every mark on my ruler so that accordingly rul[0]=0, rul[5]=5. If I'm indexing the blue circles, natural number quantities, objects, spans, makes sense to start at 1; cir= [1,2,3,4,5] so that cir[1]=1 and cir[5]=5. Now, a lot of the discussion was to do with slicing coupled with the indexing and I don't totally understand why. a ≤ x < b is not so intuitive when dealing with objects ("I want balls 1 up to the the one before 3"), so on one side, you put the finger on what you want and on the other, on what you don't want. But this method does have the neat property of producing neighbor selections that border perfectly, as in [:a][a:b][b:c]. Although in inverse order(-1), the results can be unexpected as it returns values off-by-one from its counterpart like; L=[0,1,2,3,4,5] so that L[1:3]=[1,2] and L[3:1:-1]=[3:2]. So it's consistent with the rule a ≤ x < b, grabbing the lower limit item, but it can feel strange by not producing the same selection with inverse order. a ≤ x ≤ b is a natural way to select objets ("I want the balls 1 up to 3"), so you're putting the finger on the things you want. If you inverse the order(-1) it's still very easy to grasp what are you picking because whatever you select it's included like: L=[0,1,2,3,4,5] so that L[1:3]=[1,2,3] and L[3:1:-1]=[3,2,1]. Problems seem to arrive though, when trying to do neighbor selections, where one would have to do [:a][a+1:b][b+1:c] to have the border perfectly. That terrible? Even though one could see a ≤ x < b to be more adept to 0-base, and a ≤ x ≤ b to be more adept to 1-base, the way I see it, index base and slicing rules could be somehow independent. And one would index and slice the way it would fit the rationale of the problem or the data, because even slicing a 1-base indexed array with a ≤ x < b, would still produce an expected outcome, as in cir=[1,2,3,4,5] so that cir[1:3]=[1,2] or cir[:3]=[1,2]. Same thing applying a ≤ x ≤ b to a 0-base indexed array, as in rul[0,1,2,3,4,5] so that rul[:2]=[0,1,2] or rul[0:2]=[0,1,2]. Given that python is an example of human friendly code language, emphasizing readability, wouldn't having 0 and 1 base indexing and custom slicing methods, improve the thought process when writing and reading the code, by fitting them better to specific contexts or data? Is there some language that provides both or each language picks only one? Cheers -- *------------------------------[image: http://i153.photobucket.com/albums/s202/animatics/probiner-sig.gif]Pedro Alpiarça dos Santos Animator 3DModeler Illustrator >> http://probiner.x10.mx/ <http://probiner.x10.mx/>*

8 8

PEP 484 (Type Hints) -- second draft
by Guido van Rossum March 23, 2015

March 23, 2015

Here's an updated draft for PEP 484. There's more to do (especially generic types need much more thought and writing) but I've done a bunch of editing and thinking so I think this is ready for another round of review. Remember, for technical questions it's often best to use the GitHub tracker for this PEP at https://github.com/ambv/typehinting/issues . Significant changes in this draft: - Define stubs. - Define `@overload`. - Describe `cast()`. - Fix description of `Any`. - Describe `Callable[..., t]`. - Explain why `List[t]` instead of `List<t>`. - Add section on rejected alternatives. - Various other edits for clarity. Incomplete list of TODOs: - Define and explain generics. - Covariance vs. contravariance ( https://github.com/ambv/typehinting/issues/2) - Other edits as indicated by FIXME comments. - Other edits mentioned in https://github.com/ambv/typehinting/blob/master/README.rst . Here's the full text of the PEP (hopefully it will appear soon at https://www.python.org/dev/peps/pep-0484/): PEP: 484 Title: Type Hints Version: $Revision$ Last-Modified: $Date$ Author: Guido van Rossum <guido(a)python.org>, Jukka Lehtosalo < jukka.lehtosalo(a)iki.fi>, Łukasz Langa <lukasz(a)langa.pl> Discussions-To: Python-Dev <python-dev(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 29-Sep-2014 Post-History: 16-Jan-2015,20-Mar-2015 Resolution: Abstract ======== This PEP introduces a standard syntax for type hints using annotations (PEP 3107) on function definitions. For example, here is a simple function whose argument and return type are declared in the annotations:: def greeting(name: str) -> str: return 'Hello ' + name While these annotations are available at runtime through the usual ``__annotations__`` attribute, *no type checking happens at runtime*. Instead, the proposal assumes the existence of a separate off-line type checker which users can run over their source code voluntarily. Essentially, such a type checker acts as a very powerful linter. The proposal is strongly inspired by mypy [mypy]_. For example, the type "sequence of integers" can be written as ``Sequence[int]``. The square brackets mean that no new syntax needs to be added to the language. The example here uses a custom class ``Sequence``, imported from a pure-Python module ``typing.py``. The ``Sequence[int]`` notation works by implementing ``__getitem__()`` in the metaclass. The type system supports unions, generic types, and a special type named ``Any`` which is consistent with (i.e. assignable to and from) all types. This latter feature is taken from the idea of gradual typing. Gradual typing and the full type system are explained in PEP 483. Other approaches from which we have borrowed or to which ours can be compared and contrasted are described in PEP 482. Rationale and Goals =================== PEP 3107 added support for arbitrary annotations on parts of a function definition. Although no meaning was assigned to annotations then, there has always been an implicit goal to use them for type hinting, which is listed as the first possible use case in said PEP. This PEP aims to provide a standard syntax for type annotations, opening up Python code to easier static analysis and refactoring, potential runtime type checking, and performance optimizations utilizing type information. Of these goals, static analysis is the most important. This includes support for off-line type checkers such as mypy, as well as providing a standard notation that can be used by IDEs for code completion and refactoring. Non-goals --------- While the proposed typing module will contain some building blocks for runtime type checking -- in particular a useful ``isinstance()`` implementation -- third party packages would have to be developed to implement specific runtime type checking functionality, for example using decorators or metaclasses. Using type hints for performance optimizations is left as an exercise for the reader. It should also be emphasized that Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention. Type Definition Syntax ====================== The syntax leverages PEP 3107-style annotations with a number of extensions described in sections below. In its basic form, type hinting is used by filling function annotations with classes:: def greeting(name: str) -> str: return 'Hello ' + name This denotes that the expected type of the ``name`` argument is ``str``. Analogically, the expected return type is ``str``. Subclasses of a specified argument type are also accepted as valid types for that argument. Abstract base classes, types available in the ``types`` module, and user-defined classes may be used as type hints as well. Annotations must be valid expressions that evaluate without raising exceptions at the time the function is defined. In addition, the needs of static analysis require that annotations must be simple enough to be interpreted by static analysis tools. (This is an intentionally somewhat vague requirement.) .. FIXME: Define rigorously what is/isn't supported. When used as an annotation, the expression ``None`` is considered equivalent to ``NoneType`` (i.e., ``type(None)`` for type hinting purposes. Type aliases are also valid type hints:: integer = int def retry(url: str, retry_count: integer) -> None: ... New names that are added to support features described in following sections are available in the ``typing`` package. Callbacks --------- Frameworks expecting callback functions of specific signatures might be type hinted using ``Callable[[Arg1Type, Arg2Type], ReturnType]``. Examples:: from typing import Callable def feeder(get_next_item: Callable[[], str]) -> None: # Body def async_query(on_success: Callable[[int], None], on_error: Callable[[int, Exception], None]) -> None: # Body It is possible to declare the return type of a callable without specifying the call signature by substituting a literal ellipsis (three dots) for the list of arguments:: def partial(func: Callable[..., str], *args) -> Callable[..., str]: # Body Note that there are no square brackets around the ellipsis. The arguments of the callback are completely unconstrained in this case (and keyword arguments are acceptable). Since using callbacks with keyword arguments is not perceived as a common use case, there is currently no support for specifying keyword arguments with ``Callable``. Similarly, there is no support for specifying callback signatures with a variable number of argument of a specific type. Generics -------- Since type information about objects kept in containers cannot be statically inferred in a generic way, abstract base classes have been extended to support subscription to denote expected types for container elements. Example:: from typing import Mapping, Set def notify_by_email(employees: Set[Employee], overrides: Mapping[str, str]) -> None: ... Generics can be parametrized by using a new factory available in ``typing`` called ``TypeVar``. Example:: from typing import Sequence, TypeVar T = TypeVar('T') # Declare type variable def first(l: Sequence[T]) -> T: # Generic function return l[0] In this case the contract is that the returning value is consistent with the elements held by the collection. ``TypeVar`` supports constraining parametric types to classes with any of the specified bases. Example:: from typing import Iterable X = TypeVar('X') Y = TypeVar('Y', Iterable[X]) def filter(rule: Callable[[X], bool], input: Y) -> Y: ... .. FIXME: Add an example with multiple bases defined. In the example above we specify that ``Y`` can be any subclass of Iterable with elements of type ``X``, as long as the return type of ``filter()`` will be the same as the type of the ``input`` argument. .. FIXME: Explain more about how this works. Forward references ------------------ When a type hint contains names that have not been defined yet, that definition may be expressed as a string, to be resolved later. For example, instead of writing:: def notify_by_email(employees: Set[Employee]) -> None: ... one might write:: def notify_by_email(employees: 'Set[Employee]') -> None: ... .. FIXME: Rigorously define this, and give a motivational example. Union types ----------- Since accepting a small, limited set of expected types for a single argument is common, there is a new special factory called ``Union``. Example:: from typing import Union def handle_employees(e: Union[Employee, Sequence[Employee]]) -> None: if isinstance(e, Employee): e = [e] ... A type factored by ``Union[T1, T2, ...]`` responds ``True`` to ``issubclass`` checks for ``T1`` and any of its subclasses, ``T2`` and any of its subclasses, and so on. One common case of union types are *optional* types. By default, ``None`` is an invalid value for any type, unless a default value of ``None`` has been provided in the function definition. Examples:: def handle_employee(e: Union[Employee, None]) -> None: ... As a shorthand for ``Union[T1, None]`` you can write ``Optional[T1]``; for example, the above is equivalent to:: from typing import Optional def handle_employee(e: Optional[Employee]) -> None: ... An optional type is also automatically assumed when the default value is ``None``, for example:: def handle_employee(e: Employee = None): ... This is equivalent to:: def handle_employee(e: Optional[Employee] = None) -> None: ... The ``Any`` type ---------------- A special kind of type is ``Any``. Every class is a subclass of ``Any``. This is also true for the builtin class ``object``. However, to the static type checker these are completely different. When the type of a value is ``object``, the type checker will reject almost all operations on it, and assigning it to a variable (or using it as a return value) of a more specialized type is a type error. On the other hand, when a value has type ``Any``, the type checker will allow all operations on it, and a value of type `Any`` can be assigned to a variable (or used as a return value) of a more constrained type. Platform-specific type checking ------------------------------- In some cases the typing information will depend on the platform that the program is being executed on. To enable specifying those differences, simple conditionals can be used:: from typing import PY2, WINDOWS if PY2: text = unicode else: text = str def f() -> text: ... if WINDOWS: loop = ProactorEventLoop else: loop = UnixSelectorEventLoop .. FIXME: Also define PY3 and POSIX? Arbitrary literals defined in the form of ``NAME = True`` will also be accepted by the type checker to differentiate type resolution:: DEBUG = False ... if DEBUG: class Tracer: <verbose implementation> else: class Tracer: <dummy implementation> For the purposes of type hinting, the type checker assumes ``__debug__`` is set to ``True``, in other words the ``-O`` command-line option is not used while type checking. Compatibility with other uses of function annotations ----------------------------------------------------- A number of existing or potential use cases for function annotations exist, which are incompatible with type hinting. These may confuse a static type checker. However, since type hinting annotations have no runtime behavior (other than evaluation of the annotation expression and storing annotations in the ``__annotations__`` attribute of the function object), this does not make the program incorrect -- it just makes it issue warnings when a static analyzer is used. To mark portions of the program that should not be covered by type hinting, use the following: * a ``@no_type_check`` decorator on classes and functions * a ``# type: ignore`` comment on arbitrary lines .. FIXME: should we have a module-wide comment as well? .. FIXME: suggest that other uses of annotations be replaced with decorators .. FIXME: add reference to "rejected alternatives" Type Hints on Local and Global Variables ======================================== No first-class syntax support for explicitly marking variables as being of a specific type is added by this PEP. To help with type inference in complex cases, a comment of the following format may be used:: x = [] # type: List[Employee] In the case where type information for a local variable is needed before it is declared, an ``Undefined`` placeholder might be used:: from typing import Undefined x = Undefined # type: List[Employee] y = Undefined(int) If type hinting proves useful in general, a syntax for typing variables may be provided in a future Python version. Casts ===== Occasionally the type checker may need a different kind of hint: the programmer may know that an expression is of a more constrained type than the type checker infers. For example:: from typing import List def find_first_str(a: List[object]) -> str: index = next(i for i, x in enumerate(a) if isinstance(x, str)) # We only get here if there's at least one string in a return cast(str, a[index]) The type checker infers the type ``object`` for ``a[index]``, but we know that (if the code gets to that point) it must be a string. The ``cast(t, x)`` call tells the type checker that we are confident that the type of ``x`` is ``t``. At runtime a cast always returns the expression unchanged -- it does not check the type, and it does not convert or coerce the value. Casts differ from type comments (see the previous section). When using a type comment, the type checker should still verify that the inferred type is consistent with the stated type. When using a cast, the type checker trusts the programmer. Also, casts can be used in expressions, while type comments only apply to assignments. Stub Files ========== Stub files are files containing type hints that are only for use by the type checker, not at runtime. There are several use cases for stub files: * Extension modules * 3rd party modules whose authors have not yet added type hints * Standard library modules for which type hints have not yet been written * Modules that must be compatible with Python 2 and 3 * Modules that use annotations for other purposes Stub files have the same syntax as regular Python modules. There is one feature of the ``typing`` module that may only be used in stub files: the ``@overload`` decorator described below. The type checker should only check function signatures in stub files; function bodies in stub files should just be a single ``pass`` statement. The type checker should have a configurable search path for stub files. If a stub file is found the type checker should not read the corresponding "real" module. Stub files may use the ``.py`` extension or alternatively may use the ``.pyi`` extension. The latter makes it possible to maintain stub files in the same directory as the corresponding real module. Function overloading -------------------- The ``@overload`` decorator allows describing functions that support multiple different combinations of argument types. This pattern is used frequently in builtin modules and types. For example, the ``__getitem__()`` method of the ``bytes`` type can be described as follows:: from typing import overload class bytes: ... @overload def __getitem__(self, i: int) -> int: pass @overload def __getitem__(self, s: slice) -> bytes: pass This description is more precise than would be possible using unions (which cannot express the relationship between the argument and return types):: from typing import Union class bytes: ... def __getitem__(self, a: Union[int, slice]) -> Union[int, bytes]: pass Another example where ``@overload`` comes in handy is the type of the builtin ``map()`` function, which takes a different number of arguments depending on the type of the callable:: from typing import Callable, Iterable, Iterator, Tuple, TypeVar, overload T1 = TypeVar('T1') T2 = TypeVar('T2) S = TypeVar('S') @overload def map(func: Callable[[T1], S], iter1: Iterable[T1]) -> Iterator[S]: pass @overload def map(func: Callable[[T1, T2], S], iter1: Iterable[T1], iter2: Iterable[T2]) -> Iterator[S]: pass # ... and we could add more items to support more than two iterables Note that we could also easily add items to support ``map(None, ...)``:: @overload def map(func: None, iter1: Iterable[T1]) -> Iterable[T1]: pass @overload def map(func: None, iter1: Iterable[T1], iter2: Iterable[T2]) -> Iterable[Tuple[T1, T2]]: pass The ``@overload`` decorator may only be used in stub files. While it would be possible to provide a multiple dispatch implementation using this syntax, its implementation would require using ``sys._getframe()``, which is frowned upon. Also, designing and implementing an efficient multiple dispatch mechanism is hard, which is why previous attempts were abandoned in favor of ``functools.singledispatch()``. (See PEP 443, especially its section "Alternative approaches".) In the future we may come up with a satisfactory multiple dispatch design, but we don't want such a design to be constrained by the overloading syntax defined for type hints in stub files. Exceptions ========== No syntax for listing explicitly raised exceptions is proposed. Currently the only known use case for this feature is documentational, in which case the recommendation is to put this information in a docstring. The ``typing`` Package ====================== To open the usage of static type checking to Python 3.5 as well as older versions, a uniform namespace is required. For this purpose, a new package in the standard library is introduced called ``typing``. It holds a set of classes representing builtin types with generics, namely: * Dict, used as ``Dict[key_type, value_type]`` * List, used as ``List[element_type]`` * Set, used as ``Set[element_type]``. See remark for ``AbstractSet`` below. * FrozenSet, used as ``FrozenSet[element_type]`` * Tuple, used by listing the element types, for example ``Tuple[int, int, str]``. Arbitrary-length homogeneous tuples can be expressed using one type and ellipsis, for example ``Tuple[int, ...]``. (The ``...`` here are part of the syntax.) The generic versions of concrete collection types (``Dict``, ``List``, ``Set``, ``FrozenSet``, and homogeneous arbitrary-length ``Tuple``) are mainly useful for annotating return values. For arguments, prefer the abstract collection types defined below, e.g. ``Mapping``, ``Sequence`` or ``AbstractSet``. It also introduces factories and helper members needed to express generics and union types: * Any, used as ``def get(key: str) -> Any: ...`` * Union, used as ``Union[Type1, Type2, Type3]`` * TypeVar, used as ``X = TypeVar('X', Type1, Type2, Type3)`` or simply ``Y = TypeVar('Y')`` * Undefined, used as ``local_variable = Undefined # type: List[int]`` or ``local_variable = Undefined(List[int])`` (the latter being slower during runtime) * Callable, used as ``Callable[[Arg1Type, Arg2Type], ReturnType]`` * AnyStr, equivalent to ``TypeVar('AnyStr', str, bytes)`` All abstract base classes available in ``collections.abc`` are importable from the ``typing`` package, with added generics support: * ByteString * Callable * Container * Hashable * ItemsView * Iterable * Iterator * KeysView * Mapping * MappingView * MutableMapping * MutableSequence * MutableSet * Sequence * Set as ``AbstractSet``. This name change was required because ``Set`` in the ``typing`` module means ``set()`` with generics. * Sized * ValuesView * Mapping The library includes literals for platform-specific type hinting: * PY2 * PY3, equivalent to ``not PY2`` * WINDOWS * POSIX, equivalent to ``not WINDOWS`` The following types are available in the ``typing.io`` module: * IO * BinaryIO * TextIO The following types are provided by the ``typing.re`` module: * Match and Pattern, types of ``re.match()`` and ``re.compile()`` results As a convenience measure, types from ``typing.io`` and ``typing.re`` are also available in ``typing`` (quoting Guido, "There's a reason those modules have two-letter names."). The place of the ``typing`` module in the standard library ---------------------------------------------------------- .. FIXME: complete this section (or discard?) Usage Patterns ============== The main use case of type hinting is static analysis using an external tool without executing the analyzed program. Existing tools used for that purpose like ``pyflakes`` [pyflakes]_ or ``pylint`` [pylint]_ might be extended to support type checking. New tools, like mypy [mypy]_, can be adopted specifically for this purpose. Type checking based on type hints is understood as a best-effort mechanism. In other words, whenever types are not annotated and cannot be inferred, the type checker considers such code valid. Type errors are only reported in case of explicit or inferred conflict. Moreover, as a mechanism that is not tied to execution of the code, it does not affect runtime behaviour. In other words, even in the case of a typing error, the program will continue running. The implementation of a type checker, whether linting source files or enforcing type information during runtime, is out of scope for this PEP. .. FIXME: This is somewhat redundant with the updated initial sections. .. FIXME: Describe run-time behavior of generic types. Rejected Alternatives ===================== During discussion of earlier drafts of this PEP, various objections were raised and alternatives were proposed. We discuss some of these here and explain why we reject them. Several main objections were raised. Which brackets for generic type parameters? ------------------------------------------- Most people are familiar with the use of angular brackets (e.g. ``List<int>``) in languages like C++, Java, C# and Swift to express the parametrization of generic types. The problem with these is that they are really hard to parse, especially for a simple-minded parser like Python. In most languages the ambiguities are usually dealy with by only allowing angular brackets in specific syntactic positions, where general expressions aren't allowed. (And also by using very powerful parsing techniques that can backtrack over an arbitrary section of code.) But in Python, we'd like type expressions to be (syntactically) the same as other expressions, so that we can use e.g. variable assignment to create type aliases. Consider this simple type expression:: List<int> >From the Python parser's perspective, the expression begins with the same four tokens (NAME, LESS, NAME, GREATER) as a chained comparison:: a c # I.e., (a < b) and (b > c) We can even make up an example that could be parsed both ways:: a [ c ] Assuming we had angular brackets in the language, this could be interpreted as either of the following two:: (a)[c] # I.e., (a).__getitem__(c) a ([c]) # I.e., (a < b) and (b > [c]) It would surely be possible to come up with a rule to disambiguate such cases, but to most users the rules would feel arbitrary and complex. It would also require us to dramatically change the CPython parser (and every other parser for Python). It should be noted that Python's current parser is intentionally "dumb" -- a simple grammar is easier for users to reason about. For all these reasons, square brackets (e.g. ``List[int]``) are (and have long been) the preferred syntax for generic type parameters. They can be implemented by defining the ``__getitem__()`` method on the metaclass, and no new syntax is required at all. This option works in all recent versions of Python (starting with Python 2.2). Python is not alone in this syntactic choice -- generic classes in Scala also use square brackets. What about existing uses of annotations? ---------------------------------------- One line of argument points out that PEP 3107 explicitly supports the use of arbitrary expressions in function annotations. The new proposal is then considered incompatible with the specification of PEP 3107. Our response to this is that, first of all, the current proposal does not introduce any direct incompatibilities, so programs using annotations in Python 3.4 will still work correctly and without prejudice in Python 3.5. We do hope that type hints will eventually become the sole use for annotations, but this will require additional discussion and a deprecation period after the initial roll-out of the typing module with Python 3.5. The current PEP will have provisional status (see PEP 411) until Python 3.6 is released. The fastest conceivable scheme would introduce silent deprecation of non-type-hint annotations in 3.6, full deprecation in 3.7, and declare type hints as the only allowed use of annotations in Python 3.8. This should give authors of packages that use annotations plenty of time to devise another approach, even if type hints become an overnight success. Another possible outcome would be that type hints will eventually become the default meaning for annotations, but that there will always remain an option to disable them. For this purpose the current proposal defines a decorator ``@no_type_check`` which disables the default interpretation of annotations as type hints in a given class or function. It also defines a meta-decorator ``@no_type_check_decorator`` which can be used to decorate a decorator (!), causing annotations in any function or class decorated with the latter to be ignored by the type checker. There are also ``# type: ignore`` comments, and static checkers should support configuration options to disable type checking in selected packages. Despite all these options, proposals have been circulated to allow type hints and other forms of annotations to coexist for individual arguments. One proposal suggests that if an annotation for a given argument is a dictionary literal, each key represents a different form of annotation, and the key ``'type'`` would be use for type hints. The problem with this idea and its variants is that the notation becomes very "noisy" and hard to read. Also, in most cases where existing libraries use annotations, there would be little need to combine them with type hints. So the simpler approach of selectively disabling type hints appears sufficient. The problem of forward declarations ----------------------------------- The current proposal is admittedly sub-optimal when type hints must contain forward references. Python requires all names to be defined by the time they are used. Apart from circular imports this is rarely a problem: "use" here means "look up at runtime", and with most "forward" references there is no problem in ensuring that a name is defined before the function using it is called. The problem with type hints is that annotations (per PEP 3107, and similar to default values) are evaluated at the time a function is defined, and thus any names used in an annotation must be already defined when the function is being defined. A common scenario is a class definition whose methods need to reference the class itself in their annotations. (More general, it can also occur with mutually recursive classes.) This is natural for container types, for example:: class Node: """Binary tree node.""" def __init__(self, left: Node, right: None): self.left = left self.right = right As written this will not work, because of the peculiarity in Python that class names become defined once the entire body of the class has been executed. Our solution, which isn't particularly elegant, but gets the job done, is to allow using string literals in annotations. Most of the time you won't have to use this though -- most _uses_ of type hints are expected to reference builtin types or types defined in other modules. A counterproposal would change the semantics of type hints so they aren't evaluated at runtime at all (after all, type checking happens off-line, so why would type hints need to be evaluated at runtime at all). This of course would run afoul of backwards compatibility, since the Python interpreter doesn't actually know whether a particular annotation is meant to be a type hint or something else. The double colon ---------------- A few creative souls have tried to invent solutions for this problem. For example, it was proposed to use a double colon (``::``) for type hints, solving two problems at once: disambiguating between type hints and other annotations, and changing the semantics to preclude runtime evaluation. There are several things wrong with this idea, however. * It's ugly. The single colon in Python has many uses, and all of them look familiar because they resemble the use of the colon in English text. This is a general rule of thumb by which Python abides for most forms of punctuation; the exceptions are typically well known from other programming languages. But this use of ``::`` is unheard of in English, and in other languages (e.g. C++) it is used as a scoping operator, which is a very different beast. In contrast, the single colon for type hints reads natural -- and no wonder, since it was carefully designed for this purpose (the idea long predates PEP 3107 [gvr-artima]_). It is also used in the same fashion in other languages from Pascal to Swift. * What would you do for return type annotations? * It's actually a feature that type hints are evaluated at runtime. * Making type hints available at runtime allows runtime type checkers to be built on top of type hints. * It catches mistakes even when the type checker is not run. Since it is a separate program, users may choose not to run it (or even install it), but might still want to use type hints as a concise form of documentation. Broken type hints are no use even for documentation. * Because it's new syntax, using the double colon for type hints would limit them to code that works with Python 3.5 only. By using existing syntax, the current proposal can easily work for older versions of Python 3. (And in fact mypy supports Python 3.2 and newer.) * If type hints become successful we may well decide to add new syntax in the future to declare the type for variables, for example ``var age: int = 42``. If we were to use a double colon for argument type hints, for consistency we'd have to use the same convention for future syntax, perpetuating the ugliness. Other forms of new syntax ------------------------- A few other forms of alternative syntax have been proposed, e.g. the introduction of a ``where`` keyword [roberge]_, and Cobra-inspired ``requires`` clauses. But these all share a problem with the double colon: they won't work for earlier versions of Python 3. The same would apply to a new ``__future__`` import. Other backwards compatible conventions -------------------------------------- The ideas put forward include: * A decorator, e.g. ``@typehints(name=str, returns=str)``. This could work, but it's pretty verbose (an extra line, and the argument names must be repeated), and a far cry in elegance from the PEP 3107 notation. * Stub files. We do want stub files, but they are primarily useful for adding type hints to existing code that doesn't lend itself to adding type hints, e.g. 3rd party packages, code that needs to support both Python 2 and Python 3, and especially extension modules. For most situations, having the annotations in line with the function definitions makes them much more useful. * Docstrings. There is an existing convention for docstrings, based on the Sphinx notation (``:type arg1: description``). This is pretty verbose (an extra line per parameter), and not very elegant. We could also make up something new, but the annotation syntax is hard to beat (because it was designed for this very purpose). It's also been proposed to simply wait another release. But what problem would that solve? It would just be procrastination. Is Type Hinting Pythonic? ========================= .. FIXME: Do we really need this section? Type annotations provide important documentation for how a unit of code should be used. Programmers should therefore provide type hints on public APIs, namely argument and return types on functions and methods considered public. However, because types of local and global variables can be often inferred, they are rarely necessary. The kind of information that type hints hold has always been possible to achieve by means of docstrings. In fact, a number of formalized mini-languages for describing accepted arguments have evolved. Moving this information to the function declaration makes it more visible and easier to access both at runtime and by static analysis. Adding to that the notion that “explicit is better than implicit”, type hints are indeed *Pythonic*. Acknowledgements ================ This document could not be completed without valuable input, encouragement and advice from Jim Baker, Jeremy Siek, Michael Matson Vitousek, Andrey Vlasovskikh, and Radomir Dopieralski. Influences include existing languages, libraries and frameworks mentioned in PEP 482. Many thanks to their creators, in alphabetical order: Stefan Behnel, William Edwards, Greg Ewing, Larry Hastings, Anders Hejlsberg, Alok Menghrajani, Travis E. Oliphant, Joe Pamer, Raoul-Gabriel Urma, and Julien Verlaguet. References ========== .. [mypy] http://mypy-lang.org .. [pyflakes] https://github.com/pyflakes/pyflakes/ .. [pylint] http://www.pylint.org .. [gvr-artima] http://www.artima.com/weblogs/viewpost.jsp?thread=85551 .. [roberge] http://aroberge.blogspot.com/2015/01/type-hinting-in-python-focus-on.html Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- --Guido van Rossum (python.org/~guido)

3 3

A macro for easier rich comparisons
by Petr Viktorin March 23, 2015

March 23, 2015

Hello, Rich comparison functions of many builtin types include a block of boilerplate code with a switch on the "op" argument, like this example from tupleobject: int cmp; PyObject *res; switch (op) { case Py_LT: cmp = vlen < wlen; break; case Py_LE: cmp = vlen <= wlen; break; case Py_EQ: cmp = vlen == wlen; break; case Py_NE: cmp = vlen != wlen; break; case Py_GT: cmp = vlen > wlen; break; case Py_GE: cmp = vlen >= wlen; break; default: return NULL; /* cannot happen */ } if (cmp) res = Py_True; else res = Py_False; Py_INCREF(res); return res; Each time it's implemented slightly differently: sometimes there are two values to compare, sometimes there's a cmp-style 0/1/-1 result to process; invalid "op" values are handled in various ways. Such code is also required in third-party extensions. I propose adding a public macro to ease this. My version takes two C-orderable values and the operation, similar to richcmpfunc: #define Py_RICHCOMPARE(val1, val2, op) ( \ ((op) == Py_EQ) ? PyBool_FromLong((val1) == (val2)) : \ ((op) == Py_NE) ? PyBool_FromLong((val1) != (val2)) : \ ((op) == Py_LT) ? PyBool_FromLong((val1) < (val2)) : \ ((op) == Py_GT) ? PyBool_FromLong((val1) > (val2)) : \ ((op) == Py_LE) ? PyBool_FromLong((val1) <= (val2)) : \ ((op) == Py_GE) ? PyBool_FromLong((val1) >= (val2)) : \ (Py_INCREF(Py_NotImplemented), Py_NotImplemented)) (As for the behavior for unknown op: for most cases the best thing to do is setting an error and returning NULL, but that doesn't fit into a macro. A surprising number of richcmpfunc's in CPython either return NULL without error (e.g. tupleobject), or fall through and return True/False arbitrarily (as in bytearrayobject). Datetime does an assert. I think Py_NotImplemented is a good value to return) I've made a patch with the macro and its use in CPython itself. Read it as examples of usage: I'm not saying that all these modules should start using it immediately (or ever, e.g. for speed reasons), and if they do, many could use some refactoring in other code paths. The patch is here: https://github.com/encukou/cpython/compare/master...richcmp Why I'm interested in this: When porting old code to Python 3, rich comparisons a major cause of added boilerplate – especially when wrapping a C library that exposes either trivially comparable types or "cmp" functions. With this macro, a typical trivial rich comparison function reduces to: static PyObject* mytype_richcmp(PyObject *obj1, PyObject *obj2, int op) { if (mytype_Check(obj1) && mytype_Check(obj2)) { return PY_RICHCOMPARE(mytype_get_data(obj1), mytype_get_data(obj2), op); } Py_RETURN_NOTIMPLEMENTED; } ... or with `PY_RICHCOMPARE(mytype_cmp(obj1, obj2), 0, op)`. (Note that even adding this to 3.5/3.6, would help in porting efforts: the macro itself is be easy to backport, but there are big benefits in having a standard name, known semantics, and official documentation.) Is this a PEP-worthy idea?

3 5

Warn about comparing bytes to int for `python3 -b`
by Brett Cannon March 21, 2015

March 21, 2015

One of the rather subtle issues with writing Python 2/3 code is that indexing on bytes in Python 2 returns a length-1 bytes object while in Python 3 it returns an int. Because ==/!= always returns True/False it can be a very subtle failure and tough to track down. What do people think of extending -b/-bb in Python 3 to warn when performing equality between an int and a bytes object of any length? I don't want to restrict to length-1 bytes objects because people may be doing comparisons where the result can be length-1 or any other length and thus would still have a subtle bug to pick up. Do people think this would raise a ton of false-positives? Would people find it useful?

3 5

class os.DirEntry is confusing,
by 罗勇刚(Yonggang Luo) March 21, 2015

March 21, 2015

Should be replaced with os.FsEntry or os.ScanEntry -- 此致礼罗勇刚 Yours sincerely, Yonggang Luo

2 1

Re: [Python-ideas] gofmt for Python: standardized styling as a language feature
by Guido van Rossum March 20, 2015

March 20, 2015

On Thu, Mar 19, 2015 at 2:30 PM, Terry Reedy <tjreedy(a)udel.edu> wrote: > On 3/19/2015 5:01 PM, Guido van Rossum wrote: > >> On Thu, Mar 19, 2015 at 1:40 PM, Terry Reedy <tjreedy(a)udel.edu >> > > Having the guidelines numbered (id'ed), even though still regarded >> as guidelines, could help communication. Checkers could then easily >> refer to specific guidelines. >> >> I worry this will just encourage the nit-picky attitude more. >> > > I agree that this is a possible negative. > I'll leave the rest of the discussion to other participants. It seems all possible POVs are well-represented. > Thanks for the link. I was specifically thinking of global >> renamings to satisfy PEP 8's Naming Conventions. Idlelib has a >> mishmash of module and function/method name styles. (I might look >> and see what can be done with the undocumented (except as >> 'unstable') libe2to3.) >> >> This feels hard to automate, because you don't know what names are part >> of an external specification. >> > > Modernizing idlelib names, in relation to PEP 434, is a separate > discussion. > > I think that a quick "clean up whitespace" feature would be a nice >> addition to IDLE, provided it's a user-selectable menu items (similar to >> the existing indent/dedent operations). >> > > The Format menu now has 'Strip trailing whitespace', which may be new > since you last looked. It needs to strip trailing blank lines, as > reindent.py does, in addition to trailing whitespace on each line, to make > files ready to commit to the cpython repository. Did you have in mind also > doing within-line cleanups (as a separate menu entry)? > I probably spoke out of turn (I haven't really used IDLE in years). But I could imagine another command that did something like autopep8 on a selection, making sure the whitespace around various operators is consistent with PEP 8, fixing indentation levels, and even inserting/removing blank lines in certain cases. I'm not sure I would want it to automatically break long lines, as I often have very specific ideas about the best place to break a long line, which require understanding the meaning of the code. Have you used Emacs? The python-mode written by Tim Peters and maintained by Barry Warsaw has some pretty amazing reformatting capabilities (though I rarely use them, because my fingers automatically type correctly formatted code :-). -- --Guido van Rossum (python.org/~guido)

2 1

Rewriting the build system (part 2)
by Ryan Gonzalez March 19, 2015

March 19, 2015

So...I've been looking stuff up. Don't have too much free time right now, but it was mostly Googleing. I have deemed three ideas to be best for Python: - Write a custom makefile generator. I have some ideas, but they likely won't come to light any time soon. - Use Boost.Build. It seems ugly and complicated, but it actually has a full configuring system that can detect libraries and try to compile source files. It's written in C89, not C++, and its dependency is the Boehm GC, which probably supports everywhere that Python runs on. Only issue? It's undergoing a rewrite in Python. However, at the rate it's going, Python will likely be at version 7 by the time they finish. - Stick with autotools. Trying to avoid this. Thoughts? Again? -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/

16 50

Migration of /usr/bin/python to python3
by Neil Schemenauer March 19, 2015

March 19, 2015

This has been brought up elsewhere, I think this is a better forum to discuss it. Changing /usr/bin/python directly to python3.x is bad idea, in my option. It's going to cause users trouble and the benefit to Python 3.x users is not worth it. Instead, let's try to find a smooth migration path. One idea is to install Python 3 as /usr/bin/python but have it always execute scripts with Python 2 unless they are explicitly marked as Python 3 compatible. The interpreter could parse the first line of the script to look for options. Traditionally the OS only passes the first option to the interpreter so anything after that could be used to pass 3.x specific options. For example, a script starting with #!/usr/bin/python -x -3 could signify that the script is compatible with a Python 3 interpreter. An alternative idea is to allow something like the 'encoding' declaration. It wouldn't have to be in the first line and would allow more flexibility. Many years in the future when all scripts are marked as Python 3 compatible, Python 2 can be removed from the system and /usr/bin/python will start Python 3 by default. I suppose running '/usr/bin/python' interactively could start Python 3 by default. However, I strongly feel that existing Python 2 scripts should not get broken in a flag day like way. Neil

24 73

.from and .to instead of .encode and .decode
by anatoly techtonik March 19, 2015

March 19, 2015

Hi, While looking at the code like: 'os': sysinfo['os'].decode('utf-8'), 'hostname': sysinfo['hostname'].decode('utf-8'), I can't really read if the result will be unicode or binary string in utf-8. It would be more convenient for readability to have these instead: bytes.from(encoding) -> unicode unicode.to(encoding) -> bytes .encode/.decode are confusing, because it Python 2 it was: str.encode(encoding) -> str str.decode(encoding) -> str with no encoding info attached. -- anatoly t.

8 13

Evolution of Style Guides
by Ian Lee March 18, 2015

March 18, 2015

tl;dr -- How should style guides evolve? So this is a post inspired largely by a thread that popped up today [1], and specifically Alex's message [2]. I didn't want to derail that conversation (specifically dealing with style checking / correcting as a part of the standard library), but I did want to start a separate thread for an idea that's been rattling around in my brain for a while now. (Disclaimer, I am the current lead maintainer of the pep8 package, so part of this comes from my work there.) Namely, how should style guides, and here I'm particularly looking at PEP-8, evolve? I was influenced early on in my Python learning by a number of youtube Python conference talks. In particular "A Python Æsthetic: Beauty and Why I Python" by Brandon Rhodes [3] and "Transforming Code into Beautiful, Idiomatic Python" by Raymond Hettinger [4] had quite an impact on my thinking while I was moving from C / C++ to Python a couple of years back, and I highly recommend them. Somewhat of an open question, I'm not sure I have a particularly hard and fast opinion on this issue, and I'd love to get feedback from the community. In particular, a couple of points that are worth mentioning: * There are a number of checks that are disabled in pep8 by default, due to their being somewhat controversial for many users [5] which you can read about what each of them are for in the docs [6] if you're interested. * Even within the checker community, there is some discussion about whether the tool should be rigidly set by the PEP-8 document (I tend to think it should be) and what that means for folks that don't want to follow a rule (I just disable it in my code) (see discussion about the use of ``isinstance()`` at [7]). * There are several large code bases, the Python stdlib and Twisted for starters, that don't necessarily follow PEP-8 to the letter, due to pre-dating PEP-8 or being highly specialized, or etc, and proposals to bring them more in line have received pushback over their potential for more harm than good (see [8] for an example) (which is completely valid). [1] https://mail.python.org/pipermail/python-ideas/2015-March/032545.html [2] https://mail.python.org/pipermail/python-ideas/2015-March/032556.html [3] https://www.youtube.com/watch?v=x-kB2o8sd5c [4] https://www.youtube.com/watch?v=OSGv2VnC0go [5] https://github.com/jcrocholl/pep8/blob/master/pep8.py#L68 [6] http://pep8.readthedocs.org/en/latest/intro.html#error-codes [7] https://github.com/jcrocholl/pep8/pull/313 [8] http://bugs.python.org/issue23061#msg233065 ~ Ian Lee

5 5