I don't think so. asyncio depends on selectors but not vice versa. The selectors module was not part of PEP 3156. It is solid and I don't see any reason why it should get a reprieve from the usual strict backwards compatibility standards. --Guido On Fri, Mar 14, 2014 at 6:49 PM, Eli Bendersky wrote:

On Fri, Mar 14, 2014 at 4:56 PM, Giampaolo Rodola' wrote:

...

On Fri, Mar 14, 2014 at 9:25 PM, R. David Murray wrote:

...

Not Provisional:

selectors

Wouldn't it be wiser to consider this one provisional as well?

+1 because of the intimate ties to asyncio

Eli

...

...

-- Giampaolo - http://grodola.blogspot.com

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/eliben%40gmail.com

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org

-- --Guido van Rossum (python.org/~guido)

2014-03-15 11:02 GMT+00:00 Giampaolo Rodola' :

One part which can be improved is that right now the selectors module

doesn't take advance of e/poll()'s modify() method: instead it just unregister() and register() the fd every time, which is of course considerably slower (there's also a TODO in the code about this).

I guess that can be fixed later in a safely manner.

Sure, it can be "fixed" easily, but I'd like to see the gain of this on a non-trivial benchmark (I mean a realistic workload, not just calling modify() in a tight loop).

On Sat, Mar 15, 2014 at 4:02 AM, Giampaolo Rodola' wrote:

On Sat, Mar 15, 2014 at 3:17 AM, Guido van Rossum wrote:

...

I don't think so. asyncio depends on selectors but not vice versa. The selectors module was not part of PEP 3156. It is solid and I don't see any reason why it should get a reprieve from the usual strict backwards compatibility standards.

One part which can be improved is that right now the selectors module doesn't take advance of e/poll()'s modify() method: instead it just unregister() and register() the fd every time, which is of course considerably slower (there's also a TODO in the code about this). I guess that can be fixed later in a safely manner.

Yes, it doesn't change the API so doesn't require provisional status.

Another concern I have is that we should probably rename self._epoll, self._select, self._kqueue to just "self._poller": that would make subclassing easier (see patch in issue http://bugs.python.org/issue18931) and would provide a unified interface for those users who want to reference the underlying poller object for some reason.

Oh, but I don't want to make subclassing easier! And I don't want to open up backdoors for "advanced" users. Plus, the object they'd get is different enough that they should probably know what kind of object it is. If there's a missing API on the selector, we should add that, not tell people to help themselves. There lies madness (like the monkey-patching of asyncore in the past). The only people subclassing the Selector clases should be people proposing changes to the stdlib. Otherwise, if someone wants to write a Selector-like class for a new platform as a 3rd party class, they should write it from scratch (even using copy/paste!) rather than making a subclass of one of the existing stdlib classes, because otherwise they would (perhaps unintentionally) constrained future evolution of the latter (like the optimization you proposed above). This downside of using subclassing as an API should be well known by now and widely warned against. -- --Guido van Rossum (python.org/~guido)

Maybe this would be a good subject for a series of blog posts? There is certainly plenty we have to say based on 20+ years of experience adding stuff to the stdlib (and not adding it). On Sat, Mar 15, 2014 at 8:55 AM, Nick Coghlan wrote:

On 16 March 2014 01:40, Guido van Rossum wrote:

...

On Sat, Mar 15, 2014 at 4:02 AM, Giampaolo Rodola' wrote:

This downside of using subclassing as an API should be well known by now and widely warned against.

I've actually pondered the idea of suggesting we explicitly recommend the "procedural facade around an object oriented implementation" API design model in PEP 8, but it's not really a *coding* style guide issue, and I also haven't been able to come up with a good way of summarising it.

That said, should we perhaps start codifying some of these principles as a "standard library API design guide"? We have a few additional issues to take into account that most software can ignore, like "assume that users that already know Python may be using the module and its documentation to learn a new domain, rather than the other way around, where a domain expert is just using the module to get things done". (That was the main driver for the differences between ipaddr and the accepted ipaddress API). The question about whether or not to add new boolean flags vs adding new APIs also comes up fairly often.

At the moment, there's no real way for newcomers to pick up those principles other than hanging around long enough to see them come up again.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

-- --Guido van Rossum (python.org/~guido)

On 16 March 2014 09:00, Greg Ewing wrote:

Nick Coghlan wrote:

...

On 16 March 2014 01:40, Guido van Rossum wrote:

...

This downside of using subclassing as an API should be well known by now and widely warned against.

I've actually pondered the idea of suggesting we explicitly recommend the "procedural facade around an object oriented implementation" API design model in PEP 8,

I don't think I would call this a "procedural" API. To my mind, any API that exposes objects with methods is an object-oriented API, whether it encourages subclassing those objects or not.

There are actually two variants of the approach. One is obviously procedural: you put basic data types (strings, numbers, containers, etc) in, you get basic data types out, and the fact an object managed the calculation in the middle is completely opaque to you as a user of the API. (from the outside, it still looks like a pure function, even though the internal calculation may have been highly stateful) The second variant is better exemplified by an API like contextlib.contextmanager(). That decorator returns a custom object type (now called contextlib._GeneratorContextManager). That type was originally undocumented with the name contextlib.GeneratorContextManager, and when the discrepancy was pointed out, I resolved it by adding the missing underscore (in 3.3 IIRC, could have been 3.2), rather than by documenting something I considered to be an implementation detail. The key problem this addresses is that telling people "this is a class" often overspecifies your API, because classes have a very broad interface. They support inheritance, typechecks, etc, etc. It's a really big commitment to your API users, and not one you should make lightly. Subclassing an object is also an *extremely* high coupling technique - it's genuinely difficult to refactor classes in any meaningful way with risking breakage of subclasses. As an example of unanticipated coupling issues that inhibit refactoring, consider a class where "method B" is originally written to call "method A". You later notice that this would be better structured by having both methods A and B call a new helper method C, rather than having B call A. If "method A" was a public API you now have a problem, since after the change, a subclass that previously only overrode A will now see different behaviour in method B (because that is now bypassing the override in the subclass). It can all get very, very messy, so experience has taught me that making your classes part of your public API before you're 100% certain you're happy with not only the public methods, but also the relationships between them, is a fine recipe for locking in bad design decisions in a hard to fix way. Sometimes the trade-off is worth it due to the power and flexibility that API users gain, but it needs to be recognised as the very high coupling technique that it is. By contrast, a callable API like contextlib.contextmanager just says "given a certain set of inputs, you will get some kind of object back with these characteristics, no more, no less". It is, in essence, ducktyping as applied to return types :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sat, Mar 15, 2014 at 2:44 PM, Nikolaus Rath wrote:

Guido van Rossum writes:

...

This downside of using subclassing as an API should be well known by now and widely warned against.

It wasn't known to me until now. Are these downsides described in some more detail somewhere?

So far I have always thought that, as long as I avoid using private attributes, subclassing was supported like any other use of the API.

Googling for things like "containment vs. inheritance" finds a wealth of articles explaining the issues. Apparently even the classic GoF patterns book prefers containment. The problem with subclassing is that it tends to create tighter coupling between provider and consumer of an API. I believe this is due to the addition of "protected" APIs that are supposedly okay to use for subclasses but not for other users. If you can completely avoid those I suppose you'd be okay, but note that my remark was sparked by a proposal to create just such a protected API (_poller). Additionally, a subclass and its base class share the same namespace, so a method or attribute defined by a subclass may get in the way of something with the same name added in a new version of the base class. The main area where I think subclassing is fine is when base class and subclasses are all part of the same package/module, so their evolution is controlled by the same team. I suppose there are also some good uses for abstract base classes that avoid the above issues. -- --Guido van Rossum (python.org/~guido)

Charles-François Natali writes:

2014-03-15 21:44 GMT+00:00 Nikolaus Rath :

...

Guido van Rossum writes:

...

This downside of using subclassing as an API should be well known by now and widely warned against.

It wasn't known to me until now. Are these downsides described in some more detail somewhere?

The short version is: "inheritance breaks encapsulation".

As a trivial and stupid example, let's say you need a list object which counts the number of items inserted/removed (it's completely stupid, but that's not the point :-):

So you might do something like: [...]

Very illuminating example, thanks a lot! Best, -Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C »Time flies like an arrow, fruit flies like a Banana.«