Mailman 3 September 2017 - Python-ideas

LOAD_NAME/LOAD_GLOBAL should be use getattr()
by Neil Schemenauer Sept. 14, 2017

Sept. 14, 2017

This is my idea of making module properties work. It is necessary for various lazy-loading module ideas and it cleans up the language IMHO. I think it may be possible to do it with minimal backwards compatibility problems and performance regression. To me, the main issue with module properties (or module __getattr__) is that you introduce another level of indirection on global variable access. Anywhere the module.__dict__ is used as the globals for code execution, changing LOAD_NAME/… [View More]

8 8

PEP 563 and expensive backwards compatibility
by Jim J. Jewett Sept. 14, 2017

Sept. 14, 2017

I am generally supportive of leaving the type annotations unprocessed by default, but there are use cases where they should be processed (and even cases where doing it at the right time matters, because of a side effect). I am concerned that the backwards compatibility story for non-typing cases be not just possible, but reasonable. (1) The PEP suggests opting out with @typing.no_type_hints ... The closest I could find was @typing.no_type_check, which has to be called on each object. It … [View More]

5 8

PEP 561 v2 - Packaging Static Type Information
by Ethan Smith Sept. 13, 2017

Sept. 13, 2017

Hello, V2 of my PEP on packaging type information is available at https://www.python.org/dev/peps/pep-0561/. It is also replicated below. I look forward to any suggestions or comments that people may have! Thanks Ethan ----------------------------------------------------------------------- PEP: 561 Title: Distributing and Packaging Type Information Author: Ethan Smith <ethan(a)ethanhs.me> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 09-Sep-2017 Python-… [View More]Version: 3.7 Post-History: Abstract ======== PEP 484 introduced type hints to Python, with goals of making typing gradual and easy to adopt. Currently, typing information must be distributed manually. This PEP provides a standardized means to package and distribute type information and an ordering for type checkers to resolve modules and collect this information for type checking using existing packaging architecture. Rationale ========= Currently, package authors wish to distribute code that has inline type information. However, there is no standard method to distribute packages with inline type annotations or syntax that can simultaneously be used at runtime and in type checking. Additionally, if one wished to ship typing information privately the only method would be via setting ``MYPYPATH`` or the equivalent to manually point to stubs. If the package can be released publicly, it can be added to typeshed [1]_. However, this does not scale and becomes a burden on the maintainers of typeshed. Additionally, it ties bugfixes to releases of the tool using typeshed. PEP 484 has a brief section on distributing typing information. In this section [2]_ the PEP recommends using ``shared/typehints/pythonX.Y/`` for shipping stub files. However, manually adding a path to stub files for each third party library does not scale. The simplest approach people have taken is to add ``site-packages`` to their ``MYPYPATH``, but this causes type checkers to fail on packages that are highly dynamic (e.g. sqlalchemy and Django). Specification ============= There are several motivations and methods of supporting typing in a package. This PEP recognizes three (3) types of packages that may be created: 1. The package maintainer would like to add type information inline. 2. The package maintainer would like to add type information via stubs. 3. A third party would like to share stub files for a package, but the maintainer does not want to include them in the source of the package. This PEP aims to support these scenarios and make them simple to add to packaging and deployment. The two major parts of this specification are the packaging specifications and the resolution order for resolving module type information. This spec is meant to replace the ``shared/typehints/pythonX.Y/`` spec of PEP 484 [2]_. New third party stub libraries are encouraged to distribute stubs via the third party packaging proposed in this PEP in place of being added to typeshed. Typeshed will remain in use, but if maintainers are found, third party stubs in typeshed are encouraged to be split into their own package. Packaging Type Information -------------------------- Packages must opt into supporting typing. This will be done though a distutils extension [3]_, providing a ``typed`` keyword argument to the distutils ``setup()`` command. The argument value will depend on the kind of type information the package provides. The new keyword will be added to Python 3.7 and will also be accessible in the ``typing_extensions`` package though a distutils extension. This enables a package maintainer to write :: setup( ... setup_requires=["typing_extensions"], typed="inline", ... ) Inline Typed Packages ''''''''''''''''''''' Packages that have inline type annotations simply have to pass the value ``"inline"`` to the ``typed`` argument in ``setup()``. Stub Only Packages '''''''''''''''''' For package maintainers wishing to ship stub files containing all of their type information, it is prefered that the ``*.pyi`` stubs are alongside the corresponding ``*.py`` files. However, the stubs may be put in a sub-folder of the Python sources, with the same name the ``*.py`` files are in. For example, the ``flyingcircus`` package would have its stubs in the folder ``flyingcircus/flyingcircus/``. This path is chosen so that if stubs are not found in ``flyingcircus/`` the type checker may treat the subdirectory as a normal package. The normal resolution order of checking ``*.pyi`` before ``*.py`` will be maintained. The value of the ``typed`` argument to ``setup()`` is ``"stubs"`` for this type of distribution. The author of the package is suggested to use ``package_data`` to assure the stub files are installed alongside the runtime Python code. Third Party Stub Packages ''''''''''''''''''''''''' Third parties seeking to distribute stub files are encouraged to contact the maintainer of the package about distribution alongside the package. If the maintainer does not wish to maintain or package stub files or type information inline, then a "third party stub package" should be created. The structure is similar, but slightly different from that of stub only packages. If the stubs are for the library ``flyingcircus`` then the package should be named ``flyingcircus-stubs`` and the stub files should be put in a sub-directory named ``flyingcircus``. This allows the stubs to be checked as if they were in a regular package. These packages should also pass ``"stubs"`` as the value of ``typed`` argument in ``setup()``. These packages are suggested to use ``package_data`` to package stub files. In addition, the package should indicate which version(s) of the runtime package are supported via the ``install_requires`` argument to ``setup()``. Type Checker Module Resolution Order ------------------------------------ The following is the order that type checkers supporting this PEP should resolve modules containing type information: 1. User code - the files the type checker is running on. 2. Stubs or Python source manually put in the beginning of the path. Type checkers should provide this to allow the user complete control of which stubs to use, and patch broken stubs/inline types from packages. 3. Third party stub packages - these packages can supersede the installed untyped packages. They can be found at ``pkg-stubs`` for package ``pkg``, however it is encouraged to check their metadata to confirm that they opt into type checking via the ``typed`` keyword. 4. Inline packages - finally, if there is nothing overriding the installed package, and it opts into type checking. 5. Typeshed (if used) - Provides the stdlib types and several third party libraries When resolving step (3) type checkers should assure the version of the stubs is compatible with the installed runtime package through the method described above. Type checkers that check a different Python version than the version they run on must find the type information in the ``site-packages``/``dist-packages`` of that Python version. This can be queried e.g. ``pythonX.Y -c 'import sys; print(sys.exec_prefix)'``. It is also recommended that the type checker allow for the user to point to a particular Python binary, in case it is not in the path. To check if a package has opted into type checking, type checkers are recommended to use the ``pkg_resources`` module to query the package metadata. If the ``typed`` package metadata has ``None`` as its value, the package has not opted into type checking, and the type checker should skip that package. References ========== .. [1] Typeshed (https://github.com/python/typeshed) .. [2] PEP 484, Storing and Distributing Stub Files (https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-fil…) .. [3] Distutils Extensions, Adding setup() arguments (http://setuptools.readthedocs.io/en/latest/setuptools.html#adding-setup-arg…) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: [View Less]

2 3

A reminder for PEP owners
by Guido van Rossum Sept. 13, 2017

Sept. 13, 2017

I know there's a lot of excitement around lots of new ideas. And the 3.7 feature freeze is looming (January is in a few months). But someone has to review and accept all those PEPs, and I can't do it all by myself. If you want your proposal to be taken seriously, you need to include a summary of the discussion on the mailing list (including objections, even if you disagree!) in your PEP, e.g. as an extended design rationale or under the Rejected Ideas heading. If you don't do this you risk … [View More]

2 1

PEP 563: Postponed Evaluation of Annotations, first draft
by Lukasz Langa Sept. 13, 2017

Sept. 13, 2017

PEP: 563 Title: Postponed Evaluation of Annotations Version: $Revision$ Last-Modified: $Date$ Author: Łukasz Langa <lukasz(a)langa.pl> Discussions-To: Python-Dev <python-dev(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 8-Sep-2017 Python-Version: 3.7 Post-History: Resolution: Abstract ======== PEP 3107 introduced syntax for function annotations, but the semantics were deliberately left undefined. PEP 484 introduced a standard meaning to … [View More]annotations: type hints. PEP 526 defined variable annotations, explicitly tying them with the type hinting use case. This PEP proposes changing function annotations and variable annotations so that they are no longer evaluated at function definition time. Instead, they are preserved in ``__annotations__`` in string form. This change is going to be introduced gradually, starting with a new ``__future__`` import in Python 3.7. Rationale and Goals =================== PEP 3107 added support for arbitrary annotations on parts of a function definition. Just like default values, annotations are evaluated at function definition time. This creates a number of issues for the type hinting use case: * forward references: when a type hint contains names that have not been defined yet, that definition needs to be expressed as a string literal; * type hints are executed at module import time, which is not computationally free. Postponing the evaluation of annotations solves both problems. Non-goals --------- Just like in PEP 484 and PEP 526, it should be emphasized that **Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention.** Annotations are still available for arbitrary use besides type checking. Using ``(a)typing.no_type_hints`` in this case is recommended to disambiguate the use case. Implementation ============== In a future version of Python, function and variable annotations will no longer be evaluated at definition time. Instead, a string form will be preserved in the respective ``__annotations__`` dictionary. Static type checkers will see no difference in behavior, whereas tools using annotations at runtime will have to perform postponed evaluation. If an annotation was already a string, this string is preserved verbatim. In other cases, the string form is obtained from the AST during the compilation step, which means that the string form preserved might not preserve the exact formatting of the source. Annotations need to be syntactically valid Python expressions, also when passed as literal strings (i.e. ``compile(literal, '', 'eval')``). Annotations can only use names present in the module scope as postponed evaluation using local names is not reliable. Note that as per PEP 526, local variable annotations are not evaluated at all since they are not accessible outside of the function's closure. Enabling the future behavior in Python 3.7 ------------------------------------------ The functionality described above can be enabled starting from Python 3.7 using the following special import:: from __future__ import annotations Resolving Type Hints at Runtime =============================== To resolve an annotation at runtime from its string form to the result of the enclosed expression, user code needs to evaluate the string. For code that uses type hints, the ``typing.get_type_hints()`` function correctly evaluates expressions back from its string form. Note that all valid code currently using ``__annotations__`` should already be doing that since a type annotation can be expressed as a string literal. For code which uses annotations for other purposes, a regular ``eval(ann, globals, locals)`` call is enough to resolve the annotation. The trick here is to get the correct value for globals. Fortunately, in the case of functions, they hold a reference to globals in an attribute called ``__globals__``. To get the correct module-level context to resolve class variables, use:: cls_globals = sys.modules[SomeClass.__module__].__dict__ Runtime annotation resolution and class decorators -------------------------------------------------- Metaclasses and class decorators that need to resolve annotations for the current class will fail for annotations that use the name of the current class. Example:: def class_decorator(cls): annotations = get_type_hints(cls) # raises NameError on 'C' print(f'Annotations for {cls}: {annotations}') return cls @class_decorator class C: singleton: 'C' = None This was already true before this PEP. The class decorator acts on the class before it's assigned a name in the current definition scope. The situation is made somewhat stricter when class-level variables are considered. Previously, when the string form wasn't used in annotations, a class decorator would be able to cover situations like:: @class_decorator class Restaurant: class MenuOption(Enum): SPAM = 1 EGGS = 2 default_menu: List[MenuOption] = [] This is no longer possible. Runtime annotation resolution and ``TYPE_CHECKING`` --------------------------------------------------- Sometimes there's code that must be seen by a type checker but should not be executed. For such situations the ``typing`` module defines a constant, ``TYPE_CHECKING``, that is considered ``True`` during type checking but ``False`` at runtime. Example:: import typing if typing.TYPE_CHECKING: import expensive_mod def a_func(arg: expensive_mod.SomeClass) -> None: a_var: expensive_mod.SomeClass = arg ... This approach is also useful when handling import cycles. Trying to resolve annotations of ``a_func`` at runtime using ``typing.get_type_hints()`` will fail since the name ``expensive_mod`` is not defined (``TYPE_CHECKING`` variable being ``False`` at runtime). This was already true before this PEP. Backwards Compatibility ======================= This is a backwards incompatible change. Applications depending on arbitrary objects to be directly present in annotations will break if they are not using ``typing.get_type_hints()`` or ``eval()``. Annotations that depend on locals at the time of the function/class definition are now invalid. Example:: def generate_class(): some_local = datetime.datetime.now() class C: field: some_local = 1 # NOTE: INVALID ANNOTATION def method(self, arg: some_local.day) -> None: # NOTE: INVALID ANNOTATION ... Annotations using nested classes and their respective state are still valid, provided they use the fully qualified name. Example:: class C: field = 'c_field' def method(self, arg: C.field) -> None: # this is OK ... class D: field2 = 'd_field' def method(self, arg: C.field -> C.D.field2: # this is OK ... In the presence of an annotation that cannot be resolved using the current module's globals, a NameError is raised at compile time. Deprecation policy ------------------ In Python 3.7, a ``__future__`` import is required to use the described functionality and a ``PendingDeprecationWarning`` is raised by the compiler in the presence of type annotations in modules without the ``__future__`` import. In Python 3.8 the warning becomes a ``DeprecationWarning``. In the next version this will become the default behavior. Rejected Ideas ============== Keep the ability to use local state when defining annotations ------------------------------------------------------------- With postponed evaluation, this is impossible for function locals. For classes, it would be possible to keep the ability to define annotations using the local scope. However, when using ``eval()`` to perform the postponed evaluation, we need to provide the correct globals and locals to the ``eval()`` call. In the face of nested classes, the routine to get the effective "globals" at definition time would have to look something like this:: def get_class_globals(cls): result = {} result.update(sys.modules[cls.__module__].__dict__) for child in cls.__qualname__.split('.'): result.update(result[child].__dict__) return result This is brittle and doesn't even cover slots. Requiring the use of module-level names simplifies runtime evaluation and provides the "one obvious way" to read annotations. It's the equivalent of absolute imports. Acknowledgements ================ This document could not be completed without valuable input, encouragement and advice from Guido van Rossum, Jukka Lehtosalo, and Ivan Levkivskyi. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: [View Less]

10 21

Re: [Python-ideas] LOAD_NAME/LOAD_GLOBAL should be use getattr()
by Eric Snow Sept. 13, 2017

Sept. 13, 2017

On Sep 12, 2017 10:17 AM, "Neil Schemenauer" <nas-python-ideas(a)arctrix.com> wrote: Introducing another special feature of modules to make this work is not the solution, IMHO. We should make module namespaces be more like instance namespaces. We already have a mechanism and it is getattr on objects. +1 - importlib needs to be fixed to pass modules to exec() and not dicts. From my initial experiments, it looks like importlib gets a lot simpler. Right now we pass around dicts … [View More]

2 1

lazy import via __future__ or compiler analysis
by Neil Schemenauer Sept. 13, 2017

Sept. 13, 2017

This is a half baked idea that perhaps could work. Maybe call it 2-stage module load instead of lazy. Introduce a lazy module import process that modules can opt-in to. The opt-in would either be with a __future__ statement or the compiler would statically analyze the module and determine if it is safe. E.g. if the module has no module level statements besides imports. .pyc files get some other bits of information: A) whether the module has opted for lazy import (IS_LAZY) B) the modules … [View More]

7 13

Give nonlocal the same creating power as global
by João Matos Sept. 12, 2017

Sept. 12, 2017

Hello, I would like to suggest that nonlocal should be given the same creating power as global. If I do global a_var it creates the global a_var if it doesn't exist. I think it would be great that nonlocal maintained that power. This way when I do nonlocal a_var it would create a_var in the imediate parent environment, if it didn't exist. Without nonlocal creation powers I have to create global variables or local variables after master=Tk() (in the following example): from tkinter … [View More]

5 5

Re: [Python-ideas] Hexadecimal floating literals
by Victor Stinner Sept. 12, 2017

Sept. 12, 2017

2017-09-12 1:27 GMT+02:00 Neil Schemenauer <neil(a)python.ca>: >> k = float("0x1.2492492492492p-3") # 1/7 > > Making it a different function from float() would avoid backwards > compatibility issues. I.e. float() no longer returns errors on some > inputs. In that case, I suggest float.fromhex() to remain consistent the bytes example: >>> b'123'.hex() '313233' >>> bytes.fromhex('313233') b'123' ... Oh wait, it already exists... >>> float.… [View More]

1 0

Give nonlocal the same creating power as global
by João Matos Sept. 11, 2017

Sept. 11, 2017

Hello, I would like to suggest that nonlocal should be given the same creating power as global. If I do global a_var it creates the global a_var if it doesn't exist. I think it would be great that nonlocal maintained that power. This way when I do nonlocal a_var it would create a_var in the imediate parent environment, if it didn't exist. Without nonlocal creation powers I have to create global variables or local variables after master=Tk() (in the following example): from … [View More]

1 0