I just want to summarize the status of the modules that have been added to the stdlib in 3.4 to make sure they are all labeled correctly: Provisional: asyncio pathlib Not Provisional: ensurepip enum selectors statistics tracemalloc I'm especially curious about tracemalloc, since I know Victor found something he wanted to change (add?) to the API just recently. --David
On Fri, 14 Mar 2014 16:25:56 -0400 "R. David Murray" <rdmurray@bitdance.com> wrote:
I just want to summarize the status of the modules that have been added to the stdlib in 3.4 to make sure they are all labeled correctly:
Provisional:
asyncio pathlib
These are right. Regards Antoine.
On Fri, Mar 14, 2014 at 9:25 PM, R. David Murray <rdmurray@bitdance.com>wrote:
Not Provisional:
selectors
Wouldn't it be wiser to consider this one provisional as well? -- Giampaolo - http://grodola.blogspot.com
On Fri, Mar 14, 2014 at 4:56 PM, Giampaolo Rodola' <g.rodola@gmail.com>wrote:
On Fri, Mar 14, 2014 at 9:25 PM, R. David Murray <rdmurray@bitdance.com>wrote:
Not Provisional:
selectors
Wouldn't it be wiser to consider this one provisional as well?
+1 because of the intimate ties to asyncio Eli
-- Giampaolo - http://grodola.blogspot.com
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/eliben%40gmail.com
I don't think so. asyncio depends on selectors but not vice versa. The selectors module was not part of PEP 3156. It is solid and I don't see any reason why it should get a reprieve from the usual strict backwards compatibility standards. --Guido On Fri, Mar 14, 2014 at 6:49 PM, Eli Bendersky <eliben@gmail.com> wrote:
On Fri, Mar 14, 2014 at 4:56 PM, Giampaolo Rodola' <g.rodola@gmail.com>wrote:
On Fri, Mar 14, 2014 at 9:25 PM, R. David Murray <rdmurray@bitdance.com>wrote:
Not Provisional:
selectors
Wouldn't it be wiser to consider this one provisional as well?
+1 because of the intimate ties to asyncio
Eli
-- Giampaolo - http://grodola.blogspot.com
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/eliben%40gmail.com
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On Sat, Mar 15, 2014 at 3:17 AM, Guido van Rossum <guido@python.org> wrote:
I don't think so. asyncio depends on selectors but not vice versa. The selectors module was not part of PEP 3156. It is solid and I don't see any reason why it should get a reprieve from the usual strict backwards compatibility standards.
One part which can be improved is that right now the selectors module doesn't take advance of e/poll()'s modify() method: instead it just unregister() and register() the fd every time, which is of course considerably slower (there's also a TODO in the code about this). I guess that can be fixed later in a safely manner. Another concern I have is that we should probably rename self._epoll, self._select, self._kqueue to just "self._poller": that would make subclassing easier (see patch in issue http://bugs.python.org/issue18931) and would provide a unified interface for those users who want to reference the underlying poller object for some reason. -- Giampaolo - http://grodola.blogspot.com
2014-03-15 11:02 GMT+00:00 Giampaolo Rodola' <g.rodola@gmail.com>:
One part which can be improved is that right now the selectors module
doesn't take advance of e/poll()'s modify() method: instead it just unregister() and register() the fd every time, which is of course considerably slower (there's also a TODO in the code about this).
I guess that can be fixed later in a safely manner.
Sure, it can be "fixed" easily, but I'd like to see the gain of this on a non-trivial benchmark (I mean a realistic workload, not just calling modify() in a tight loop).
On 15 March 2014 21:02, Giampaolo Rodola' <g.rodola@gmail.com> wrote:
On Sat, Mar 15, 2014 at 3:17 AM, Guido van Rossum <guido@python.org> wrote:
I don't think so. asyncio depends on selectors but not vice versa. The selectors module was not part of PEP 3156. It is solid and I don't see any reason why it should get a reprieve from the usual strict backwards compatibility standards.
One part which can be improved is that right now the selectors module doesn't take advance of e/poll()'s modify() method: instead it just unregister() and register() the fd every time, which is of course considerably slower (there's also a TODO in the code about this). I guess that can be fixed later in a safely manner.
Another concern I have is that we should probably rename self._epoll, self._select, self._kqueue to just "self._poller": that would make subclassing easier (see patch in issue http://bugs.python.org/issue18931) and would provide a unified interface for those users who want to reference the underlying poller object for some reason.
Those are currently private APIs regardless, so provisional or stable doesn't make a difference to our ability to change them. Provisional status is designed to cover cases like asyncio or the 3.3 ipaddress module, where we think they're largely ready for inclusion in the standard library, but aren't yet prepared to offer full backwards compatibility guarantees due to the potential discovery of issues in some of the more obscure edge cases. It's still preferred to skip that step if the contributors/maintainers of a new module are confident in the level of vetting the API has already received (the smaller and more targeted the API, the easier it is to skip an initial provisional release, especially when incorporating an existing PyPI project that doesn't require any significant API changes to be suitable for the stdlib). The experience with ipaddress (where we flipped it to stable for 3.4 with only changes that could be counted as bug fixes anyway) is still the one we expect to be typical for provisional modules. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Mar 15, 2014 at 4:02 AM, Giampaolo Rodola' <g.rodola@gmail.com>wrote:
On Sat, Mar 15, 2014 at 3:17 AM, Guido van Rossum <guido@python.org>wrote:
I don't think so. asyncio depends on selectors but not vice versa. The selectors module was not part of PEP 3156. It is solid and I don't see any reason why it should get a reprieve from the usual strict backwards compatibility standards.
One part which can be improved is that right now the selectors module doesn't take advance of e/poll()'s modify() method: instead it just unregister() and register() the fd every time, which is of course considerably slower (there's also a TODO in the code about this). I guess that can be fixed later in a safely manner.
Yes, it doesn't change the API so doesn't require provisional status.
Another concern I have is that we should probably rename self._epoll, self._select, self._kqueue to just "self._poller": that would make subclassing easier (see patch in issue http://bugs.python.org/issue18931) and would provide a unified interface for those users who want to reference the underlying poller object for some reason.
Oh, but I don't want to make subclassing easier! And I don't want to open up backdoors for "advanced" users. Plus, the object they'd get is different enough that they should probably know what kind of object it is. If there's a missing API on the selector, we should add that, not tell people to help themselves. There lies madness (like the monkey-patching of asyncore in the past). The only people subclassing the Selector clases should be people proposing changes to the stdlib. Otherwise, if someone wants to write a Selector-like class for a new platform as a 3rd party class, they should write it from scratch (even using copy/paste!) rather than making a subclass of one of the existing stdlib classes, because otherwise they would (perhaps unintentionally) constrained future evolution of the latter (like the optimization you proposed above). This downside of using subclassing as an API should be well known by now and widely warned against. -- --Guido van Rossum (python.org/~guido)
On 16 March 2014 01:40, Guido van Rossum <guido@python.org> wrote:
On Sat, Mar 15, 2014 at 4:02 AM, Giampaolo Rodola' <g.rodola@gmail.com> wrote:
This downside of using subclassing as an API should be well known by now and widely warned against.
I've actually pondered the idea of suggesting we explicitly recommend the "procedural facade around an object oriented implementation" API design model in PEP 8, but it's not really a *coding* style guide issue, and I also haven't been able to come up with a good way of summarising it. That said, should we perhaps start codifying some of these principles as a "standard library API design guide"? We have a few additional issues to take into account that most software can ignore, like "assume that users that already know Python may be using the module and its documentation to learn a new domain, rather than the other way around, where a domain expert is just using the module to get things done". (That was the main driver for the differences between ipaddr and the accepted ipaddress API). The question about whether or not to add new boolean flags vs adding new APIs also comes up fairly often. At the moment, there's no real way for newcomers to pick up those principles other than hanging around long enough to see them come up again. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Maybe this would be a good subject for a series of blog posts? There is certainly plenty we have to say based on 20+ years of experience adding stuff to the stdlib (and not adding it). On Sat, Mar 15, 2014 at 8:55 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 16 March 2014 01:40, Guido van Rossum <guido@python.org> wrote:
On Sat, Mar 15, 2014 at 4:02 AM, Giampaolo Rodola' <g.rodola@gmail.com> wrote:
This downside of using subclassing as an API should be well known by now and widely warned against.
I've actually pondered the idea of suggesting we explicitly recommend the "procedural facade around an object oriented implementation" API design model in PEP 8, but it's not really a *coding* style guide issue, and I also haven't been able to come up with a good way of summarising it.
That said, should we perhaps start codifying some of these principles as a "standard library API design guide"? We have a few additional issues to take into account that most software can ignore, like "assume that users that already know Python may be using the module and its documentation to learn a new domain, rather than the other way around, where a domain expert is just using the module to get things done". (That was the main driver for the differences between ipaddr and the accepted ipaddress API). The question about whether or not to add new boolean flags vs adding new APIs also comes up fairly often.
At the moment, there's no real way for newcomers to pick up those principles other than hanging around long enough to see them come up again.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-- --Guido van Rossum (python.org/~guido)
Nick Coghlan wrote:
On 16 March 2014 01:40, Guido van Rossum <guido@python.org> wrote:
This downside of using subclassing as an API should be well known by now and widely warned against.
I've actually pondered the idea of suggesting we explicitly recommend the "procedural facade around an object oriented implementation" API design model in PEP 8,
I don't think I would call this a "procedural" API. To my mind, any API that exposes objects with methods is an object-oriented API, whether it encourages subclassing those objects or not. -- Greg
On 16 March 2014 09:00, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Nick Coghlan wrote:
On 16 March 2014 01:40, Guido van Rossum <guido@python.org> wrote:
This downside of using subclassing as an API should be well known by now and widely warned against.
I've actually pondered the idea of suggesting we explicitly recommend the "procedural facade around an object oriented implementation" API design model in PEP 8,
I don't think I would call this a "procedural" API. To my mind, any API that exposes objects with methods is an object-oriented API, whether it encourages subclassing those objects or not.
There are actually two variants of the approach. One is obviously procedural: you put basic data types (strings, numbers, containers, etc) in, you get basic data types out, and the fact an object managed the calculation in the middle is completely opaque to you as a user of the API. (from the outside, it still looks like a pure function, even though the internal calculation may have been highly stateful) The second variant is better exemplified by an API like contextlib.contextmanager(). That decorator returns a custom object type (now called contextlib._GeneratorContextManager). That type was originally undocumented with the name contextlib.GeneratorContextManager, and when the discrepancy was pointed out, I resolved it by adding the missing underscore (in 3.3 IIRC, could have been 3.2), rather than by documenting something I considered to be an implementation detail. The key problem this addresses is that telling people "this is a class" often overspecifies your API, because classes have a very broad interface. They support inheritance, typechecks, etc, etc. It's a really big commitment to your API users, and not one you should make lightly. Subclassing an object is also an *extremely* high coupling technique - it's genuinely difficult to refactor classes in any meaningful way with risking breakage of subclasses. As an example of unanticipated coupling issues that inhibit refactoring, consider a class where "method B" is originally written to call "method A". You later notice that this would be better structured by having both methods A and B call a new helper method C, rather than having B call A. If "method A" was a public API you now have a problem, since after the change, a subclass that previously only overrode A will now see different behaviour in method B (because that is now bypassing the override in the subclass). It can all get very, very messy, so experience has taught me that making your classes part of your public API before you're 100% certain you're happy with not only the public methods, but also the relationships between them, is a fine recipe for locking in bad design decisions in a hard to fix way. Sometimes the trade-off is worth it due to the power and flexibility that API users gain, but it needs to be recognised as the very high coupling technique that it is. By contrast, a callable API like contextlib.contextmanager just says "given a certain set of inputs, you will get some kind of object back with these characteristics, no more, no less". It is, in essence, ducktyping as applied to return types :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Guido van Rossum <guido@python.org> writes:
This downside of using subclassing as an API should be well known by now and widely warned against.
It wasn't known to me until now. Are these downsides described in some more detail somewhere? So far I have always thought that, as long as I avoid using private attributes, subclassing was supported like any other use of the API. Best, Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C »Time flies like an arrow, fruit flies like a Banana.«
On Sat, Mar 15, 2014 at 2:44 PM, Nikolaus Rath <Nikolaus@rath.org> wrote:
Guido van Rossum <guido@python.org> writes:
This downside of using subclassing as an API should be well known by now and widely warned against.
It wasn't known to me until now. Are these downsides described in some more detail somewhere?
So far I have always thought that, as long as I avoid using private attributes, subclassing was supported like any other use of the API.
Googling for things like "containment vs. inheritance" finds a wealth of articles explaining the issues. Apparently even the classic GoF patterns book prefers containment. The problem with subclassing is that it tends to create tighter coupling between provider and consumer of an API. I believe this is due to the addition of "protected" APIs that are supposedly okay to use for subclasses but not for other users. If you can completely avoid those I suppose you'd be okay, but note that my remark was sparked by a proposal to create just such a protected API (_poller). Additionally, a subclass and its base class share the same namespace, so a method or attribute defined by a subclass may get in the way of something with the same name added in a new version of the base class. The main area where I think subclassing is fine is when base class and subclasses are all part of the same package/module, so their evolution is controlled by the same team. I suppose there are also some good uses for abstract base classes that avoid the above issues. -- --Guido van Rossum (python.org/~guido)
2014-03-15 21:44 GMT+00:00 Nikolaus Rath <Nikolaus@rath.org>:
Guido van Rossum <guido@python.org> writes:
This downside of using subclassing as an API should be well known by now and widely warned against.
It wasn't known to me until now. Are these downsides described in some more detail somewhere?
The short version is: "inheritance breaks encapsulation". As a trivial and stupid example, let's say you need a list object which counts the number of items inserted/removed (it's completely stupid, but that's not the point :-): So you might do something like: """ class CountingList(list): [...] def append(self, e): self.inserted += 1 return super().append(e) def extend(self, l): self.inserted += len(l) return super().extend(l) """ Looks fine, it would probably work. Now, it's actually very fragile: imagine what would happen if list.extend() was internally implemented by calling list.append() for each element: you'd end up counting each element twice (since the subclass append() method would be called). And that's the problem: by deriving from a class, you become dependent of its implementation, even though you're using its public API. Which means that it could work with e.g. CPython but not Pypy, or break with a new version of Python. Another related problem is, as Guido explained, that if you add a new method in the subclass, and the parent class gains a method with the same name in a new version, you're in trouble. That's why advising inheritance as a silver bullet for code reuses is IMO one of the biggest mistakes in OOP, simply because although attractive, inheritance breaks encapsulation. As a rule of thumb, you should only use inheritance within a module/package, or in other words only if you're in control of the implementation. The alternative is to use "composition" For more details, I highly encourage anyone interested in looking at the book "Effective Java" by Joshua Bloch (the example above is inspired by his book). Although Java-centric, it's packed with many advises, patterns and anti-patterns that are relevant to OOP and just programming in general (it's in my top-5 books). cf
Charles-François Natali <cf.natali@gmail.com> writes:
2014-03-15 21:44 GMT+00:00 Nikolaus Rath <Nikolaus@rath.org>:
Guido van Rossum <guido@python.org> writes:
This downside of using subclassing as an API should be well known by now and widely warned against.
It wasn't known to me until now. Are these downsides described in some more detail somewhere?
The short version is: "inheritance breaks encapsulation".
As a trivial and stupid example, let's say you need a list object which counts the number of items inserted/removed (it's completely stupid, but that's not the point :-):
So you might do something like: [...]
Very illuminating example, thanks a lot! Best, -Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C »Time flies like an arrow, fruit flies like a Banana.«
On Mar 14, 2014, at 4:25 PM, R. David Murray <rdmurray@bitdance.com> wrote:
I just want to summarize the status of the modules that have been added to the stdlib in 3.4 to make sure they are all labeled correctly:
Provisional:
asyncio pathlib
Not Provisional:
ensurepip
ensurepip is correct.
enum selectors statistics tracemalloc
I'm especially curious about tracemalloc, since I know Victor found something he wanted to change (add?) to the API just recently.
--David _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
I'm especially curious about tracemalloc, since I know Victor found something he wanted to change (add?) to the API just recently.
I hope that the PEP process found all major design issues. I will try to avoid as much as posssible to break the backward compatibility. As you wrote, I may add new functions, methods or features. You can add the "provisional API" if you want. It would allow to fix the API if a design issue is found. Victor
participants (12)
-
Antoine Pitrou -
Charles-François Natali -
Donald Stufft -
Eli Bendersky -
Ethan Furman -
Giampaolo Rodola' -
Greg Ewing -
Guido van Rossum -
Nick Coghlan -
Nikolaus Rath -
R. David Murray -
Victor Stinner