Mailman 3 November 2017 - Python-ideas

A proliferation of (un-)Pythonically programmatic pragmas
by Barry Warsaw 20 Nov '17

20 Nov '17

I love many of the ancillary tools that help improve the quality of my Python code, including flake8, coverage, and mypy. Each of these usually produces some great diagnostics, but none of them are perfect, so they also produce false positives that have to be suppressed. Each of these tools supports "pragmas", i.e. comments you can add to a line of code to suppress a warning. E.g. with flake8 I can say: import readline, rlcompleter # noqa: F401 to suppress the flake8 warnings about unused imports. These modules work by side-effect, which of course a static analyzer can't know. Similarly, when mypy complains about a line it's confused on, I can add a comment to suppress the useless warning: try: from pathlib import Path, PurePath except ImportError: from pathlib2 import Path, PurePath # type: ignore And when I want to boost coverage, but I know that some lines aren't covered under some versions of Python, I can do something like: self.send_error(HTTPStatus.NOT_FOUND) # pragma: nocover These are all well and good until I have to *combine* suppressions. E.g. in the pathlib2 pragma to mypy, I also get a flake8 warning and I've tried just about every combination of pragma comment I can think of, but I've not been able to make both tools happy. I've resorted to refactoring the code into an entire module for flake8 to ignore and added: # flake8: noqa to suppress all warnings in the file. I actually have hit on a few places where I need to suppress warnings from all three tools on the same line, and I basically can't do it. The specifics aren't as important as the general use case: multiple tools competing for the same valuable real-estate. I have no ideas how to improve the situation, and of course any solution would involve some coordination between all of these tools, but it's beginning to feel like a losing battle. Is there a better way? Cheers, -Barry

13 21

Terminology of types / typing [was: PEP 560 (second post)]
by Koos Zevenhoven 15 Nov '17

15 Nov '17

Here are some thoughts––maybe even a proposal––for type-related terminology, because clear terminology makes discussion and reasoning easier, and helps avoid errors. (And related to the PEP 560 thread, the question of what should go into class attributes like __bases__). Terminology regarding types is confusing enough, and of all the terminology I've seen, I like these most: * concrete type: something that concretely implements data storage or functionality. Usually this is a normal class. * abstract type: an assumption or set of assumptions made about an instance. While both kinds can (and probably should) have a corresponding runtime representation, there's not that much that we can currently assume about the second kind––abstract types. They could be almost anything. Anyway, regarding PEP 560, in my speculated naming, the __bases__ attribute of a class would contain both concrete and abstract bases. Those with any concrete method implementations should go in the mro. But then there's the problem that "abstract base classes" may contain both concrete and abstract methods. What do we call such a "type"? Maybe we have both "concrete" and "strictly concrete" types. Perhaps we also have both "abstract" and "strictly abstract" types. An ABC with some concrete default implementations might then be both a concrete type and an abstract type. Note that in the above bullet point "definition" of concrete type, I intentionally left out the requirement that the type can be instantiated. The other two bullet points are: * strictly concrete type: a concrete type that is not abstract––it concretely implements everything that it represents / describes. This is almost always a normal class, so it might be also known as "class". * strictly abstract type: an abstract type that is not concrete––it does not implement any functionality or storage. There might be a way to improve terminology from this, but I feel that what I sketched here is usable but still not very ambiguous. ––Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven +

2 3

Greetings from a lurker
by Robert 15 Nov '17

15 Nov '17

Hi, I've been lurking for quite a time and just wanted to let you know I'm out here. Thanks for making Python the best! Robert Kaplan

1 0

Modules as global namespaces rather than dicts
by Neil Schemenauer 15 Nov '17

15 Nov '17

This is an idea I have been playing with and seems to hold some promise. I think we should use a module instance as the standard global namespace rather than directly using its dict. I have a prototype version of CPython that does this, not working 100% yet though. Major changes: - In the frameobject, remove f_builtins and f_globals, add f_namespace that refers to the module. Make f_globals and f_builtins into properties. - Change ceval to use f_namespace, rather than carrying around globals and builtins. Change functions that take globals as a dict so they also accept a module object. - Change the module object to keep track of the builtins for it. - Change funcobject (e.g. PyFunction_NewWithQualName) to accept 'globals' as a module object - When given a dict and we now expect a module object, create an anonymous module to wrap it. This part is a bit tricky due to reference cycles and speed requirements. However, my hope is that if the internals of CPython can be made to pass modules instead of dicts around, these anonymous modules will become a rare case and so won't matter if they are a little slower. So, what is the purpose of all this trouble? - I believe quite a lot of Python internals can be simpler. For example, importlib is complicated by the fact that a dict is passed around when most of the logic would prefer to have the module. Grubbing in sys.modules to lookup the module object is ugly. The expression "exec(code, module)" is elegant to me. - I believe it will be possible to make global variable access work similar to fast locals. I.e. have each module contain a indexed list of global names, and use an array lookup rather than a dict lookup in the normal case. For backwards compatibility, my idea is to keep a flag on the module object noting if fast global behavior is okay. If someone grabs the module dict, e.g. using module.__dict__ , vars() or frame.f_globals then we clear the flag and revert to the slow dict behavior. Also, if a piece of code is executed in a different namespace, we have to revert to the slow case. That does not seem so hard to do. - I want to have properties for module globals and have them work from within the module. I.e. changing something from a global variable to a property should not require changes to other module code. I.e. LOAD_GLOBAL should trigger a property get. Regards, Neil

4 4

PEP 560 (second post)
by Ivan Levkivskyi 15 Nov '17

15 Nov '17

Previously I posted PEP 560 two weeks ago, while several other PEPs were also posted, so it didn't get much of attention. Here I post the PEP 560 again, now including the full text for convenience of commenting. -- Ivan ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ PEP: 560 Title: Core support for generic types Author: Ivan Levkivskyi <levkivskyi(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 03-Sep-2017 Python-Version: 3.7 Post-History: 09-Sep-2017 Abstract ======== Initially PEP 484 was designed in such way that it would not introduce *any* changes to the core CPython interpreter. Now type hints and the ``typing`` module are extensively used by the community, e.g. PEP 526 and PEP 557 extend the usage of type hints, and the backport of ``typing`` on PyPI has 1M downloads/month. Therefore, this restriction can be removed. It is proposed to add two special methods ``__class_getitem__`` and ``__subclass_base__`` to the core CPython for better support of generic types. Rationale ========= The restriction to not modify the core CPython interpreter lead to some design decisions that became questionable when the ``typing`` module started to be widely used. There are three main points of concerns: performance of the ``typing`` module, metaclass conflicts, and the large number of hacks currently used in ``typing``. Performance: ------------ The ``typing`` module is one of the heaviest and slowest modules in the standard library even with all the optimizations made. Mainly this is because subscripted generic types (see PEP 484 for definition of terms used in this PEP) are class objects (see also [1]_). The three main ways how the performance can be improved with the help of the proposed special methods: - Creation of generic classes is slow since the ``GenericMeta.__new__`` is very slow; we will not need it anymore. - Very long MROs for generic classes will be twice shorter; they are present because we duplicate the ``collections.abc`` inheritance chain in ``typing``. - Time of instantiation of generic classes will be improved (this is minor however). Metaclass conflicts: -------------------- All generic types are instances of ``GenericMeta``, so if a user uses a custom metaclass, then it is hard to make a corresponding class generic. This is particularly hard for library classes that a user doesn't control. A workaround is to always mix-in ``GenericMeta``:: class AdHocMeta(GenericMeta, LibraryMeta): pass class UserClass(LibraryBase, Generic[T], metaclass=AdHocMeta): ... but this is not always practical or even possible. With the help of the proposed special attributes the ``GenericMeta`` metaclass will not be needed. Hacks and bugs that will be removed by this proposal: ----------------------------------------------------- - ``_generic_new`` hack that exists since ``__init__`` is not called on instances with a type differing form the type whose ``__new__`` was called, ``C[int]().__class__ is C``. - ``_next_in_mro`` speed hack will be not necessary since subscription will not create new classes. - Ugly ``sys._getframe`` hack, this one is particularly nasty, since it looks like we can't remove it without changes outside ``typing``. - Currently generics do dangerous things with private ABC caches to fix large memory consumption that grows at least as O(N\ :sup:`2`), see [2]_. This point is also important because it was recently proposed to re-implement ``ABCMeta`` in C. - Problems with sharing attributes between subscripted generics, see [3]_. Current solution already uses ``__getattr__`` and ``__setattr__``, but it is still incomplete, and solving this without the current proposal will be hard and will need ``__getattribute__``. - ``_no_slots_copy`` hack, where we clean-up the class dictionary on every subscription thus allowing generics with ``__slots__``. - General complexity of the ``typing`` module, the new proposal will not only allow to remove the above mentioned hacks/bugs, but also simplify the implementation, so that it will be easier to maintain. Specification ============= The idea of ``__class_getitem__`` is simple: it is an exact analog of ``__getitem__`` with an exception that it is called on a class that defines it, not on its instances, this allows us to avoid ``GenericMeta.__getitem__`` for things like ``Iterable[int]``. The ``__class_getitem__`` is automatically a class method and does not require ``@classmethod`` decorator (similar to ``__init_subclass__``) and is inherited like normal attributes. For example:: class MyList: def __getitem__(self, index): return index + 1 def __class_getitem__(cls, item): return f"{cls.__name__}[{item.__name__}]" class MyOtherList(MyList): pass assert MyList()[0] == 1 assert MyList[int] == "MyList[int]" assert MyOtherList()[0] == 1 assert MyOtherList[int] == "MyOtherList[int]" Note that this method is used as a fallback, so if a metaclass defines ``__getitem__``, then that will have the priority. If an object that is not a class object appears in the bases of a class definition, the ``__subclass_base__`` is searched on it. If found, it is called with the original tuple of bases as an argument. If the result of the call is not ``None``, then it is substituted instead of this object. Otherwise (if the result is ``None``), the base is just removed. This is necessary to avoid inconsistent MRO errors, that are currently prevented by manipulations in ``GenericMeta.__new__``. After creating the class, the original bases are saved in ``__orig_bases__`` (currently this is also done by the metaclass). NOTE: These two method names are reserved for exclusive use by the ``typing`` module and the generic types machinery, and any other use is strongly discouraged. The reference implementation (with tests) can be found in [4]_, the proposal was originally posted and discussed on the ``typing`` tracker, see [5]_. Backwards compatibility and impact on users who don't use ``typing``: ===================================================================== This proposal may break code that currently uses the names ``__class_getitem__`` and ``__subclass_base__``. This proposal will support almost complete backwards compatibility with the current public generic types API; moreover the ``typing`` module is still provisional. The only two exceptions are that currently ``issubclass(List[int], List)`` returns True, with this proposal it will raise ``TypeError``. Also ``issubclass(collections.abc.Iterable, typing.Iterable)`` will return ``False``, which is probably desirable, since currently we have a (virtual) inheritance cycle between these two classes. With the reference implementation I measured negligible performance effects (under 1% on a micro-benchmark) for regular (non-generic) classes. References ========== .. [1] Discussion following Mark Shannon's presentation at Language Summit (https://github.com/python/typing/issues/432) .. [2] Pull Request to implement shared generic ABC caches (https://github.com/python/typing/pull/383) .. [3] An old bug with setting/accessing attributes on generic types (https://github.com/python/typing/issues/392) .. [4] The reference implementation (https://github.com/ilevkivskyi/cpython/pull/2/files) .. [5] Original proposal (https://github.com/python/typing/issues/468) Copyright ========= This document has been placed in the public domain.

6 18

Re: [Python-ideas] Looking for input to help with the pip situation
by Michel Desmoulin 15 Nov '17

15 Nov '17

General summary so far ####################### This debate has been very civil so far and quite productive, so thanks to everyone involved. Several issues and solution proposals have been mentioned, so I will make a summary here. Issue 1: running Python is a different command different OS when several versions are installed ================================================================================================== Proposal A: ----------- Suffix Python executable on Windows like on Unix, so that people will type pythonX.X if they want a specify version. Pros: easy and discoverable. Cons: you need a lot of stuff in the system path. Proposal B: ------------ Provide the "py" command on Unix, so that people will type "py -x.x" Pros: no more issues with system path. Just add this command, and register python installations to be listed by "py". Cons: very few people know about "py", even on windows. This would need lots of communication. Harder to change Mac and majors Linux distros setup than just the next windows installer. Issue 2: there is no one and only one obvious way to do venv / pip =================================================================== Because pip and venv are tied to a Python version, there are numerous ways to call them. Just for pip, you have "pip", "py -x.x -m pip", "pythonx.x -m pip" and "pipx". Proposal A: ----------- Decide on one form, and promote it everywhere. Pros: simple. Cons: long and tedious process, with a huge inertia before all 3rd party to it. Would need a "PEP 8 for Python command". Requires "issue 1" to be solved before. Proposal B: ------------ Provide a pipenv-like behavior pipenv is a tool created by the author of "requests" that automatically "do the right thing" by default. The first time you install a package, it creates a virtualenv for the current directory (but don't store it in the current directory). It will then install all packages in there. "pipenv shell" starts a shell in the virtualenv. Dependencies are automatically dumped in a requirement files separating dev and prod. Pros: bypass the multiple installers issue by prompting you what version you wish the first time it creates a virtualenv. Solve the problem of admin rights and system path. Cons: lots of work. Either a new tool or breaking compat with pip. If debian didn't install pip by default, this may not help. Issue 3: pip and virtualenv are not available everywhere =================================================================== Proposal A: ------------ Make a mock "pip" commands for when it's not installed. venv already does that in some cases (see attachements), alhought it does give you the wrong command so it should be fixed. The gist of it: if the users try to run pip install while it's missing, it prints a warning explaining it's not installed and the command you can run to solve the problem plus a link to more information in the documentation. Pros: low friction to implement. Cons: requires to put up a hook somewhere so that each linux distro can input "the proper commands to use" for their system when they package python. More a workaround than a solution. Proposal B: ------------ Make pip and venv part of the standard and request debian that they provide it. Pros: straight forward. Cons: holy war in the making. ................. Can I have +1 and -1 from each of you on each point ? Should I start a new thread for each of them ? Favorite personnal combination ================================ 1 - So far, I like "providing py on unix" better, but I think it's too hard to do. So provide suffix on windows seems more achievable. So +1 for having pythonX.X on windows. 2 - Decide on one form, and promote it everywhere is again the simplest IMO. I love pipenv but the strings to pull to get something like this means it probably would not happen. If we stick to suffix for issue 1, this mean we can promote: pythonX.X -m pip pythonX.X -m venv Everywhere. In the docs. In the tutorials. In teaching classes. This will help a lot to remove all confusions and make the Python start up experience way easier. 3 - getting angry debian devs is probably not wise. The "mock" pip/venv command also plays very well with the 2 other solutions. It means we can promote everywhere: pythonX.X -m cmd It will work most of the time. And when it doesn't, give clear contextual steps on

5 6

PEP 562
by Ivan Levkivskyi 14 Nov '17

14 Nov '17

I have written a short PEP as a complement/alternative to PEP 549. I will be grateful for comments and suggestions. The PEP should appear online soon. -- Ivan *********************************************************** PEP: 562 Title: Module __getattr__ Author: Ivan Levkivskyi <levkivskyi(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 09-Sep-2017 Python-Version: 3.7 Post-History: 09-Sep-2017 Abstract ======== It is proposed to support ``__getattr__`` function defined on modules to provide basic customization of module attribute access. Rationale ========= It is sometimes convenient to customize or otherwise have control over access to module attributes. A typical example is managing deprecation warnings. Typical workarounds are assigning ``__class__`` of a module object to a custom subclass of ``types.ModuleType`` or substituting ``sys.modules`` item with a custom wrapper instance. It would be convenient to simplify this procedure by recognizing ``__getattr__`` defined directly in a module that would act like a normal ``__getattr__`` method, except that it will be defined on module *instances*. For example:: # lib.py from warnings import warn deprecated_names = ["old_function", ...] def _deprecated_old_function(arg, other): ... def __getattr__(name): if name in deprecated_names: warn(f"{name} is deprecated", DeprecationWarning) return globals()[f"_deprecated_{name}"] raise AttributeError(f"module {__name__} has no attribute {name}") # main.py from lib import old_function # Works, but emits the warning There is a related proposal PEP 549 that proposes to support instance properties for a similar functionality. The difference is this PEP proposes a faster and simpler mechanism, but provides more basic customization. An additional motivation for this proposal is that PEP 484 already defines the use of module ``__getattr__`` for this purpose in Python stub files, see [1]_. Specification ============= The ``__getattr__`` function at the module level should accept one argument which is a name of an attribute and return the computed value or raise an ``AttributeError``:: def __getattr__(name: str) -> Any: ... This function will be called only if ``name`` is not found in the module through the normal attribute lookup. The reference implementation for this PEP can be found in [2]_. Backwards compatibility and impact on performance ================================================= This PEP may break code that uses module level (global) name ``__getattr__``. The performance implications of this PEP are minimal, since ``__getattr__`` is called only for missing attributes. References ========== .. [1] PEP 484 section about ``__getattr__`` in stub files (https://www.python.org/dev/peps/pep-0484/#stub-files) .. [2] The reference implementation (https://github.com/ilevkivskyi/cpython/pull/3/files) Copyright ========= This document has been placed in the public domain.

10 17

Complete typing.Union with the rest of set-like functions
by Ilya Kulakov 13 Nov '17

13 Nov '17

I needed to declare a type that would mean "Any but None" and didn't find how. Current implementation allows to expand set of allowed types, not narrow it. Perhaps typing needs the rest of set operators in addition to Union? Best Regards, Ilya Kulakov

2 1

Any chance on (slowly) deprecating `eval` and `exec` as builtins?
by אלעזר 09 Nov '17

09 Nov '17

Hi, The dangers of eval and exec are obvious and well known to advanced users, but the availability as built-in functions makes it too tempting for beginners or even medium-level programmers. You can see questions about these function pretty often in stackoverflow (roughly once a day <https://stackoverflow.com/search?tab=newest&q=eval%20python>, though sometimes the uses are legitimate). Maybe we could start a ten-year process of deprecating the use of `builtins.eval` (in the docs, and then with warnings)? `builtins.eval` will be a wrapper to the real evaluation function, moved to `unsafe.eval` or something obvious like that, so all you need to do to port your code is to add `from unsafe import unsafe_eval as eval, unsafe_exec as exec` at the top of the file; it will be a nice warning to the reader. The fact that it is a wrapper will slightly slow it down and make the stack traces noisier - both are good things, IMO. Also, it is unfortunate that `ast.literal_eval` is less accessible than `builtins.eval`. Giving it an alias in builtins might make it easier for programmers (and less scary - "ast" might sound like I need a PhD to use it). What do you think? Elazar

10 15

Possible enhancement to typing
by Steve Barnes 08 Nov '17

08 Nov '17

The discussions on moving typing to a standalone library prompted me to take a look at the documentation for typing which in turn lead to an idea that should be reasonably simple to implement but greatly add value to typing for some applications. If a group of iterators were to be added to the typing module it would be reasonably simple to automatically add and assert to any decorated modules to ensure that such modules were always called with the documented types. I am thinking of decorators such as: - @typing.assert_params_mismatch - this would provide a wrapper that had auto-generated asserts that all the parameters were of designated types. - @typing.debug_assert_params_mismatch - this would provide a wrapper that had auto-generated asserts that all the parameters were of designated types only if a DEBUG environmental variable was set or similar. - @typing.warn_params_mismatch - this would provide a wrapper that had auto-generated warnings issued to stderr if any of the parameters were of not of the designated types. - @typing.coerce_params - this would add a wrapper to attempt to convert any mismatched parameters to the specified type(s). I also think that this might increase the uptake of typing by giving some clear benefits outside of documentation and static type checking. Pushing this out for comment before considering a more formal, (PEP), proposal. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com

4 8