Mailman 3 March 2016 - Python-ideas

Is multiple inheritance Pythonic?
by Martin Teichmann March 19, 2016

March 19, 2016

Hi List, Summary: Multiple inheritance is not used in new code of the standard library anymore, even where it could be of good use. Should multiple inheritance thus be abandoned, or supported more? I am currently developing a little library for the communication in our project. It is supposed to support different protocols, encryptions and addressing standards. At first I thought I'd just use multiple inheritance. A user of the library would then just write a little stub-class, as in: … [View More]class MyProtocol(URLAdressMixin, SSLEncryptionMixin, HTTPProtocol): pass and voilà! everything is ready. But because I try to be one of the cool guys, I tried to do all of that using asyncio. It has a pluggable event loop, where you can plug in your own event loop with the specifics of your project. Again, I thought, let's to multiple inheritance! I needed some priority scheduling, so just write a mixin that does that, and mix it into the base class of the event loop. To my astonishment, the designers of asyncio explicitly decided against that option. They prefer not to let users subclass at all. Instead, asyncio has a factory plugin system, as an example, one can call set_task_factory to modify what the BaseEventLoop.create_task method is doing. Things like that used to be clearly the domain of mixin classes. The question arised, why would one do that? Is multiple inheritance bad, or not Pythonic? Guessing what the decisions of the asyncio developers were, several problems come to my mind. Firstly, metaclasses a notoriously problematic in a multiple inheritance context. Some of you might have seen my work on PEP 487 to mitigate that problem. Another issue are the stub classes mentioned above. One ends up writing lots of little stub classes combining mixins together. (An example from https://github.com/jupyter/qtconsole/blob/master/qtconsole/inprocess.py: class QtInProcessKernelManager(QtKernelManagerMixin, InProcessKernelManager):) Many users don't like that, it's too much programming, too little using. Another problem is that mixins cannot be mixed-in at runtime anymore. Once the event loop is running, I cannot sneak in a new task factory anymore. It is questionable whether that's desired. Another argument is that using the factory way, a library might "sneak in" its factories without the user noticing. But in my opinion, that's against the Pythonic concept of being explicit. A library should rather document: "Hey, you need to include SuperTaskMixin into your Eventloop", rather than silently setting a new factory. This also touches mainainability a lot. Imagine you are using two libraries, both at some time deciding they need to set a task factory. That won't work out. All of this is nicely solved in a multiple inheritance scheme: the method resolution order using super() is flexible enough to allow two independent extensions of create_task, as long as it is properly documented how to use it. So both concepts, multiple inheritance as well as pluggable factories, have their pros and cons. According to the Zen, there should ideally only be one way of doing things, but given that I'm not Dutch I just don't see what that way is. Please help me out. In my opinion, either a) multiple inheritance should be declared not Pythonic and frowned upon, or b) made easy to use and well supported In case of a), it would be nice to have proper facilities set up to have a good factory plugin mechanism (for example one that also allows to put in several plugins) Or in the b) case, it should be easier to use multiple inheritance and mixins. I was thinking about a class algebra, so following the above example one could create a mixed class simply as MyProtocol = SSLMixin + HTTPProtocol. This would lead do options (going back to asyncio) like asyncio.set_event_loop((PriorityMixin + QtEventLoop)()) What do you all think about that? Greetings Martin [View Less]

5 7

Make py.exe default to Python 3
by Paul Moore March 18, 2016

March 18, 2016

The "py.exe" wrapper on Windows runs the latest Python version available on the user's PC, but it prefers Python 2 if the user has both Python 2 and 3 installed. With Python 3.5 being released, is it not about time that we switched that default to always prefer the latest version available? For scripts, users can always specify a shebang line, exactly as on Unix. For interactive use, people who prefer Python 2 simply have to create a py.ini file saying [defaults] python=2 At the moment, … [View More]

22 73

Adding python to Windows. Was :Re: Make py.exe default to Python 3
by Oscar Benjamin March 17, 2016

March 17, 2016

On 16 Mar 2016 17:47, "Steve Dower" <steve.dower(a)python.org> wrote: > > The uninstall problem is actually that "python" would stop working completely in that case, and it's not necessarily obvious why or how to fix it. > > The leave-behind problem doesn't really matter as the directory is not searched if it doesn't exist, but we also don't leave behind entries added by the installer. A user who customizes their own PATH will obviously need to manually remove the entry, but … [View More]

4 4

Re: [Python-ideas] How the heck does async/await work in Python 3.5
by Sven R. Kunze March 14, 2016

March 14, 2016

On 20.02.2016 07:53, Christian Gollwitzer wrote: > If you have difficulties wit hthe overall concept, and if you are open > to discussions in another language, take a look at this video: > > https://channel9.msdn.com/Shows/C9-GoingNative/GoingNative-39-await-co-rout… > > > MS has added coroutine support with very similar syntax to VC++ > recently, and the developer tries to explain it to the "stackful" > programmers. Because of this thread, I finally finished an … [View More]

9 15

Map-then-filter in comprehensions
by Allan Clark March 12, 2016

March 12, 2016

tl;dr What is support like for adding an 'as' clause to comprehension syntax? In order to allow map-then-filter, it might look like something this: [y for x in numbers if abs(x) as y > 5] I wish to propose an extension to Python comprehension syntax in an attempt to make it applicable in more areas. I'll first describe the deficiency I perceive in the current comprehension syntax and then propose my extension. I'll then note some drawbacks. For the purposes of illustration I'll use … [View More]list comprehension syntax but I believe everything I say is equally applicable to set and dictionary comprehensions. I'll also talk about lists but again (more or less) everything applies to the more general concept of iterables. Comprehensions are essentially a filter followed by a map operation. So if you need to perform a filter operation followed by a map operation over a list, this is pretty convenient in Python. Here we are going to take the absolute value of all even numbers within the range -10 to 10. numbers = range(-10, 10) [abs(x) for x in numbers if x % 2 == 0] However, if you wish to perform a map operation and *then* a filter operation this is not so convenient, so suppose we wish to obtain the absolute value of all numbers that have an absolute value larger than 5, we can do this by calling the mapped method twice: abs(x) for x in numbers if abs(x) > 5] This is a bit unsatisfying and even impossible in the case that the mapped method has some side-effect (although arguably if you find yourself in that situation you have taken a mis-step somewhere). An alternative is to apply the mapping first: [y for y in (abs(x) for x in numbers) if y > 5] I have to say I quite like this, as it is pretty explicit, but it is a bit unsatisfying that you require only one comprehension for a filter-then-map but two for a map-then-filter. What would be nice is if we could give a name to the mapped expression and then use that in the filter. [abs(x) as y for x in numbers if y > 5] I don't like this as it means the order of execution is dependent on whether there is an 'as' clause, in particular the 'if' clause itself may do some computation such as in `if f(y) > 5`. An altenative is to allow 'as' expressions in the condition, something like: [y for x in numbers if abs(x) as y > 5] Note that it would be possible to map to an intermediate result, such as: [y**2 for x in numbers if abs(x) as y > 5] Alternatively we could put the 'as' in the pattern: [y**2 for abs(x) as y in numbers if y > 5] I did not like this as it is obscures the fact that 'x' is being set to each element of 'numbers'. Additionally, we might later wish to adopt a functional programming idiom in which we use 'as' for deconstructive assignment whilst giving a name to the entire matched value, for example: [p for (x,y) as p if x > y] Or more generally: (x,y) as a = f(z) But that is getting somewhat off-topic. I promised some drawbacks: * I am confident there are some implementation gotchas nestling in here somewhere. * I could imagine how abuse of such a mechanism to lead to pretty unreadable code. * I'm still not *that* upset by the explicit map first: `[y for y in (abs(x) for x in numbers) if y > 5]` * I could see how it is not immediately obvious what the code does. * It would need to be decided whether you allowed multiple 'as' expression in the condition, particularly using 'and' or 'or' as in 'if f(a) as x > 5 and f(b) as y > 5' To summarise: * It's a touch annoying that comprehensions allow filter-then-map but not map-then-filter * Three proposed syntaxes are: * [abs(x) as y for x in numbers if y > 5] * [y for x in numbers if abs(x) as y > 5] * [y**2 for abs(x) as y in numbers if y > 5] * My favourite is the middle one. Finally these seem to currently be syntax errors so we should not break any existing code. [View Less]

14 25

Version control system link in package meta data
by Tymoteusz Jankowski March 12, 2016

March 12, 2016

Hi, tldr: install project and its requirements as cloned repositories (if possible) Let's say I'm developing requests <https://github.com/kennethreitz/requests> library which relies on these <https://github.com/kennethreitz/requests/blob/master/requirements.txt> packages. My workflow is this: $ git clone https://github.com/kennethreitz/requests.git $ cd requests $ virtualenv requests $ . requests/bin/activate.fish $ pip install -r requirements.txt Now I can change Requests … [View More]

2 1

FW: Map-then-filter in comprehensions
by Émanuel Barry March 10, 2016

March 10, 2016

Hello, I’m mostly in favour of extending the comprehension syntax, but not at any cost. The `as` proposal is a big -1 from me. Right now, `as` is used for : - from foo import bar as baz - try: ... except Exception as e: ... And maybe a few others I’ve probably forgotten. The general concept is name binding, so if I see something like [x for x, y in some_iterable as y > 5] I’m going be confused by what sort of name binding it does. A quick glance at the keywords … [View More]

12 23

Pseudo-package for current directory
by Chris Angelico March 9, 2016

March 9, 2016

Every once in a while, the issue pops up again of "import X" picking up X.py from the current directory, when a stdlib or pip-installed module named X was wanted. Python almost has a mechanism for protecting against this, in the form of explicit package relative imports; the only problem is that the current directory isn't a package. So the solution is to treat the current directory as a pseudo-package. It'd be a backward-incompatible change, so it would need to be explicitly invoked. … [View More]

9 17

Thoughts on lambda expressions
by Abe Dillon March 3, 2016

March 3, 2016

I'm new here, but I know that lambda expression syntax has probably been discussed to oblivion. I have searched previous posts without finding anything exactly like my idea so here it is: # instead of this hand = sorted(cards, key=lambda card: card.suit) # sort some iterable of cards by suit # do this hand = sorted(cards, key=(card.suit from card)) # sort some iterable of cards by suit # |<= you can stop reading here and still have a good sense of … [View More]

13 32

Simpler Customization of Class Creation - PEP 487
by Martin Teichmann March 2, 2016

March 2, 2016

Hi List, about a year ago I started a discussion on how to simplify metaclasses, which led to PEP 487. I got some good ideas from this list, but couldn't follow up on this because I was bound in other projects. In short, metaclasses are often not used as they are considered very complicated. Indeed they are, especially if you need to use two of them at the same time in a multiple inheritance context. Most metaclasses, however, serve only some of the following three purposes: a) run some code … [View More]after a class is created b) initialize descriptors of a class or c) keep the order in which class attributes have been defined. PEP 487 now proposes to put a metaclass into the standard library, which can be used for all those three purposes. If now libraries start to use this metaclass, we won't need any metaclass mixing anymore. What has changed since the last time I posted PEP 487? Firstly, I re-wrote large parts of the PEP to make it easier to read. Those who liked the old text, that's still existing in PEP 422. Secondly, I modified the proposal following suggestions from this list: I added the descriptor initialization (purpose b)), as this was considered particularly useful, even if it could in principle be done using purpose a) from above. The order-keeping of the class attributes is the leftover from a much more ambitious previous idea that would have allowed for custom namespaces during class creation. But this additional feature would have rendered the most common usecase - getting the order of attributes - much more complicated, so I opted for usability over flexibility. I have put the new version of the PEP here: https://github.com/tecki/metaclasses/blob/pep487/pep-0487.txt and also added it to this posting. An implementation of this PEP can be found at: https://pypi.python.org/pypi/metaclass Greetings Martin PEP: 487 Title: Simpler customisation of class creation Version: $Revision$ Last-Modified: $Date$ Author: Martin Teichmann <lkb.teichmann(a)gmail.com>, Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 27-Feb-2015 Python-Version: 3.6 Post-History: 27-Feb-2015, 5-Feb-2016 Replaces: 422 Abstract ======== Currently, customising class creation requires the use of a custom metaclass. This custom metaclass then persists for the entire lifecycle of the class, creating the potential for spurious metaclass conflicts. This PEP proposes to instead support a wide range of customisation scenarios through a new ``__init_subclass__`` hook in the class body, a hook to initialize descriptors, and a way to keep the order in which attributes are defined. Those hooks should at first be defined in a metaclass in the standard library, with the option that this metaclass eventually becomes the default ``type`` metaclass. The new mechanism should be easier to understand and use than implementing a custom metaclass, and thus should provide a gentler introduction to the full power Python's metaclass machinery. Background ========== Metaclasses are a powerful tool to customize class creation. They have, however, the problem that there is no automatic way to combine metaclasses. If one wants to use two metaclasses for a class, a new metaclass combining those two needs to be created, typically manually. This need often occurs as a surprise to a user: inheriting from two base classes coming from two different libraries suddenly raises the necessity to manually create a combined metaclass, where typically one is not interested in those details about the libraries at all. This becomes even worse if one library starts to make use of a metaclass which it has not done before. While the library itself continues to work perfectly, suddenly every code combining those classes with classes from another library fails. Proposal ======== While there are many possible ways to use a metaclass, the vast majority of use cases falls into just three categories: some initialization code running after class creation, the initalization of descriptors and keeping the order in which class attributes were defined. Those three use cases can easily be performed by just one metaclass. If this metaclass is put into the standard library, and all libraries that wish to customize class creation use this very metaclass, no combination of metaclasses is necessary anymore. The three use cases are achieved as follows: 1. The metaclass contains an ``__init_subclass__`` hook that initializes all subclasses of a given class, 2. the metaclass calls an ``__init_descriptor__`` hook for all descriptors defined in the class, and 3. an ``__attribute_order__`` tuple is left in the class in order to inspect the order in which attributes were defined. For ease of use, a base class ``SubclassInit`` is defined, which uses said metaclass and contains an empty stub for the hook described for use case 1. As an example, the first use case looks as follows:: class SpamBase(SubclassInit): # this is implicitly a @classmethod def __init_subclass__(cls, **kwargs): # This is invoked after a subclass is created, but before # explicit decorators are called. # The usual super() mechanisms are used to correctly support # multiple inheritance. # **kwargs are the keyword arguments to the subclasses' # class creation statement super().__init_subclass__(cls, **kwargs) class Spam(SpamBase): pass # the new hook is called on Spam The base class ``SubclassInit`` contains an empty ``__init_subclass__`` method which serves as an endpoint for cooperative multiple inheritance. Note that this method has no keyword arguments, meaning that all methods which are more specialized have to process all keyword arguments. This general proposal is not a new idea (it was first suggested for inclusion in the language definition `more than 10 years ago`_, and a similar mechanism has long been supported by `Zope's ExtensionClass`_), but the situation has changed sufficiently in recent years that the idea is worth reconsidering for inclusion. The second part of the proposal adds an ``__init_descriptor__`` initializer for descriptors. Descriptors are defined in the body of a class, but they do not know anything about that class, they do not even know the name they are accessed with. They do get to know their owner once ``__get__`` is called, but still they do not know their name. This is unfortunate, for example they cannot put their associated value into their object's ``__dict__`` under their name, since they do not know that name. This problem has been solved many times, and is one of the most important reasons to have a metaclass in a library. While it would be easy to implement such a mechanism using the first part of the proposal, it makes sense to have one solution for this problem for everyone. To give an example of its usage, imagine a descriptor representing weak referenced values (this is an insanely simplified, yet working example):: import weakref class WeakAttribute: def __get__(self, instance, owner): return instance.__dict__[self.name] def __set__(self, instance, value): instance.__dict__[self.name] = weakref.ref(value) # this is the new initializer: def __init_descriptor__(self, owner, name): self.name = name The third part of the proposal is to leave a tuple called ``__attribute_order__`` in the class that contains the order in which the attributes were defined. This is a very common usecase, many libraries use an ``OrderedDict`` to store this order. This is a very simple way to achieve the same goal. Key Benefits ============ Easier inheritance of definition time behaviour ----------------------------------------------- Understanding Python's metaclasses requires a deep understanding of the type system and the class construction process. This is legitimately seen as challenging, due to the need to keep multiple moving parts (the code, the metaclass hint, the actual metaclass, the class object, instances of the class object) clearly distinct in your mind. Even when you know the rules, it's still easy to make a mistake if you're not being extremely careful. Understanding the proposed implicit class initialization hook only requires ordinary method inheritance, which isn't quite as daunting a task. The new hook provides a more gradual path towards understanding all of the phases involved in the class definition process. Reduced chance of metaclass conflicts ------------------------------------- One of the big issues that makes library authors reluctant to use metaclasses (even when they would be appropriate) is the risk of metaclass conflicts. These occur whenever two unrelated metaclasses are used by the desired parents of a class definition. This risk also makes it very difficult to *add* a metaclass to a class that has previously been published without one. By contrast, adding an ``__init_subclass__`` method to an existing type poses a similar level of risk to adding an ``__init__`` method: technically, there is a risk of breaking poorly implemented subclasses, but when that occurs, it is recognised as a bug in the subclass rather than the library author breaching backwards compatibility guarantees. A path of introduction into Python ================================== Most of the benefits of this PEP can already be implemented using a simple metaclass. For the ``__init_subclass__`` hook this works all the way down to Python 2.7, while the attribute order needs Python 3.0 to work. Such a class has been `uploaded to PyPI`_. The only drawback of such a metaclass are the mentioned problems with metaclasses and multiple inheritance. Two classes using such a metaclass can only be combined, if they use exactly the same such metaclass. This fact calls for the inclusion of such a class into the standard library, let's call it ``SubclassMeta``, with the base class using it called ``SubclassInit``. Once all users use this standard library metaclass, classes from different packages can easily be combined. But still such classes cannot be easily combined with other classes using other metaclasses. Authors of metaclasses should bear that in mind and inherit from the standard metaclass if it seems useful for users of the metaclass to add more functionality. Ultimately, if the need for combining with other metaclasses is strong enough, the proposed functionality may be introduced into Python's ``type``. Those arguments strongly hint to the following procedure to include the proposed functionality into Python: 1. The metaclass implementing this proposal is put onto PyPI, so that it can be used and scrutinized. 2. Once the code is properly mature, it can be added to the Python standard library. There should be a new module called ``metaclass`` which collects tools for metaclass authors, as well as a documentation of the best practices of how to write metaclasses. 3. If the need of combining this metaclass with other metaclasses is strong enough, it may be included into Python itself. While the metaclass is still in the standard library and not in the language, it may still clash with other metaclasses. The most prominent metaclass in use is probably ABCMeta. It is also a particularly good example for the need of combining metaclasses. For users who want to define a ABC with subclass initialization, we should support a ``ABCSubclassInit`` class, or let ABCMeta inherit from this PEP's metaclass. Extensions written in C or C++ also often define their own metaclass. It would be very useful if those could also inherit from the metaclass defined here, but this is probably not possible. New Ways of Using Classes ========================= This proposal has many usecases like the following. In the examples, we still inherit from the ``SubclassInit`` base class. This would become unnecessary once this PEP is included in Python directly. Subclass registration --------------------- Especially when writing a plugin system, one likes to register new subclasses of a plugin baseclass. This can be done as follows:: class PluginBase(SubclassInit): subclasses = [] def __init_subclass__(cls, **kwargs): super().__init_subclass__(**kwargs) cls.subclasses.append(cls) One should note that this also works nicely as a mixin class. Trait descriptors ----------------- There are many designs of Python descriptors in the wild which, for example, check boundaries of values. Often those "traits" need some support of a metaclass to work. This is how this would look like with this PEP:: class Trait: def __get__(self, instance, owner): return instance.__dict__[self.key] def __set__(self, instance, value): instance.__dict__[self.key] = value def __init_descriptor__(self, owner, name): self.key = name class Int(Trait): def __set__(self, instance, value): # some boundary check code here super().__set__(instance, value) Rejected Design Options ======================= Calling the hook on the class itself ------------------------------------ Adding an ``__autodecorate__`` hook that would be called on the class itself was the proposed idea of PEP 422. Most examples work the same way or even better if the hook is called on the subclass. In general, it is much easier to explicitly call the hook on the class in which it is defined (to opt-in to such a behavior) than to opt-out, meaning that one does not want the hook to be called on the class it is defined in. This becomes most evident if the class in question is designed as a mixin: it is very unlikely that the code of the mixin is to be executed for the mixin class itself, as it is not supposed to be a complete class on its own. The original proposal also made major changes in the class initialization process, rendering it impossible to back-port the proposal to older Python versions. Other variants of calling the hook ---------------------------------- Other names for the hook were presented, namely ``__decorate__`` or ``__autodecorate__``. This proposal opts for ``__init_subclass__`` as it is very close to the ``__init__`` method, just for the subclass, while it is not very close to decorators, as it does not return the class. Requiring an explicit decorator on ``__init_subclass__`` -------------------------------------------------------- One could require the explicit use of ``@classmethod`` on the ``__init_subclass__`` decorator. It was made implicit since there's no sensible interpretation for leaving it out, and that case would need to be detected anyway in order to give a useful error message. This decision was reinforced after noticing that the user experience of defining ``__prepare__`` and forgetting the ``@classmethod`` method decorator is singularly incomprehensible (particularly since PEP 3115 documents it as an ordinary method, and the current documentation doesn't explicitly say anything one way or the other). Defining arbitrary namespaces ----------------------------- PEP 422 defined a generic way to add arbitrary namespaces for class definitions. This approach is much more flexible than just leaving the definition order in a tuple. The ``__prepare__`` method in a metaclass supports exactly this behavior. But given that effectively the only use cases that could be found out in the wild were the ``OrderedDict`` way of determining the attribute order, it seemed reasonable to only support this special case. The metaclass described in this PEP has been designed to be very simple such that it could be reasonably made the default metaclass. This was especially important when designing the attribute order functionality: This was a highly demanded feature and has been enabled through the ``__prepare__`` method of metaclasses. This method can be abused in very weird ways, making it hard to correctly maintain this feature in CPython. This is why it has been proposed to deprecated this feature, and instead use ``OrderedDict`` as the standard namespace, supporting the most important feature while dropping most of the complexity. But this would have meant that ``OrderedDict`` becomes a language builtin like dict and set, and not just a standard library class. The choice of the ``__attribute_order__`` tuple is a much simpler solution to the problem. A more ``__new__``-like hook ---------------------------- In PEP 422 the hook worked more like the ``__new__`` method than the ``__init__`` method, meaning that it returned a class instead of modifying one. This allows a bit more flexibility, but at the cost of much harder implementation and undesired side effects. History ======= This used to be a competing proposal to PEP 422 by Nick Coughlan and Daniel Urban. It shares both most of the PEP text and proposed code, but has major differences in how to achieve its goals. In the meantime, PEP 422 has been withdrawn favouring this approach. References ========== .. _published code: http://mail.python.org/pipermail/python-dev/2012-June/119878.html .. _more than 10 years ago: http://mail.python.org/pipermail/python-dev/2001-November/018651.html .. _Zope's ExtensionClass: http://docs.zope.org/zope_secrets/extensionclass.html .. _uploaded to PyPI: https://pypi.python.org/pypi/metaclass Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: [View Less]

7 16