Where to put non-collection ABCs (was: Deprecating the old-style sequence protocol)

Speaking of using ABCs more, where should we put ABCs which have nothing to do with collections? As of right now all ABCs seem to get shoved into collections.abc, but e.g. Awaitable and Coroutine are not types of collections. I personally want to add a context manager ABC with a default __exit__. I opened http://bugs.python.org/issue25637 to discuss this, but I figured a wider discussion wouldn't hurt. Some suggest just putting the ABCs into the abc module. We could create an interfaces module (top-level or a submodule of ABC). The other option is to put the ABCs in subject-specific modules, so my context manager one would go into contextlib (either top-level or an abc submodule); don't know where the coroutine ones would go since it might be overloading asyncio if we out them there. Anyway, the key point is collections.abc is starting to get non-collections stuff and if we are going to start pushing ABCs more we should decide how we want to organize them in general in the stdlib and instead of dumping them into collections.abc. On Sun, Dec 27, 2015, 09:35 Guido van Rossum <guido@python.org> wrote:
_______________________________________________

On 28.12.2015 18:42, Brett Cannon wrote:
I'd put them into the abc module (perhaps turning this into a package, if things get too crowded). collections.abc could then do a "from abc import *" for b/w compatibility.
-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Dec 28 2015)
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/

On 29 December 2015 at 03:58, M.-A. Lemburg <mal@egenix.com> wrote:
With the benefit of hindsight, I think a broad namespace separation like that might have been a good way to go from a data model discoverability perspective, as it clearly separates the abstract descriptions of data and control flow modelling concepts from the concrete implementations of those concepts. However, we should also keep in mind which standard library modules already publish ABCs, which is at least: typing io numbers collections.abc selectors email (That list was collected by grepping Python files for ABCMeta, so I may have missed some) That suggests to me that this design decision has effectively already been made, and it's to include the ABCs in the relevant domain specific modules. The inclusion of the non-collections related ABCs in collections.abc are the anomaly, which can be addressed by moving them out to a more appropriate location (adjusting the documentation accordingly), and then importing them into collections.abc for backwards compatibility (taking care not to increase the startup import footprint too much in the process). The current collections.abc interfaces which I think are most at issue here: Callable Iterable Iterator Generator Awaitable Coroutine Awaitable AsyncIterable AsyncIterator (I'm excluding Hashable from the list, as the main reason that matters is in describing whether or not something is suitable for inclusion in a dict or set) These differ from the rest of the collections.abc interfaces in that they're more closely associated with language level control flow syntax than they are with containers specifically: Callable - function calls Iterable - for loops, comprehensions Iterator - for loops, comprehensions Generator - generators, generator expressions Awaitable - await expressions Coroutine - async def AsyncIterable - async for AsyncIterator - async for Adding ContextManager and AsyncContextManager would give ABCs for the protocols related to "with" and "async with". Since these all correspond to syntactic protocols, I now think it's reasonable to include them directly in the "abc" namespace, since that still gives us a clear guideline for which ABCs go there, and which belong somewhere else: if it has syntax associated with it, or it's part of the ABC machinery itself, then it can go directly in the "abc" namespace. Using the "abc" namespace also ensures there isn't any additional import overhead, since that gets imported by all ABC using code anyway. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

With MAL, Nick, and Yury all suggesting the syntax-related ABCs go into the abc module and no one else really suggesting otherwise beyond me, I think it's reasonable to go with the syntax rule-of-thumb for what ABCs go into abc and all other abstract concepts go into domain-specific modules (e.g., import-related ones go into importlib.abc, which Nick forgot to list :). On Mon, 28 Dec 2015 at 19:58 Nick Coghlan <ncoghlan@gmail.com> wrote:

Would it be a good idea to mix 'concrete implementations of ABCs'* directly in the abc module where the tooling to create ABCs live, or to put it in a submodule? I feel it should be a submodule, but that isn't based on vast experience. * Yes, I know, these are not concrete implementations of types... I find it a little confusing to describe. On 12/28/2015 22:58, Nick Coghlan wrote:

On Thu, Dec 31, 2015 at 12:48 PM Brett Cannon <brett@python.org> wrote:
Locating collections ABCs in a submodule makes some sense, as there are 21 of them and the collections module is important for beginners to learn without getting distracted by ABCs. Contrast that with the direct inclusion of ABCs in most other modules and it suggests the creation of a submodule for collections may have been motivated for the same reason as this discussion -- it didn't feel right to have certain ABCs directly in the collections module. If the non-collection ABCs are being moved out of the collections module and into the ``abc`` module, there's less reason to separate them into a submodule. Beginners don't venture into the abc module expecting to understand everything. It's natural to find a bunch of ABCs in a module called ``abc``. And ABCs are included directly in many other modules instead of being relegated to a less discoverable submodule like ``typing.abc``, ``io.abc``, ``numbers.abc``, etc. as many of those are focused on ABCs in the first place. It's easy to notice a submodule when reading docs on the internet, but it's hard to figure out what the correct module is to import when hanging out at a basic REPL. Flat is better than nested and all that. * Yes, I know, these are not concrete implementations of types... I find

On 1 January 2016 at 08:18, Michael Selik <mike@selik.org> wrote:
No need to speculate, the original motive for the move is documented in the tracker: http://bugs.python.org/issue11085 (which can be found by looking at the commit history for the collections module: https://hg.python.org/cpython/log/3.5/Lib/collections/abc.py ) The problem was with folks getting confused between the abstract types like Sequence and Mapping and the usable classes like deque, ChainMap, OrderedDict, defaultdict, etc, rather than there being a lot of non-collections related content in the file. At the time, Callable was the only non-container related ABC in collections.abc - most of the others now being considered for relocation (Generator, Coroutine, Awaitable, AsyncIterable, AsyncIterator) were added as part of the PEP 492 implementation in 3.5, and *that* was mostly driven by Iterable and Iterator already being there so it was "logical" to also add Generator, AsyncIterable and AsyncIterator, with Coroutine and Awaitable coming along for the ride. That does raise the question of whether or not it's worth continuing to publish the PEP 492 ABCs from collections.abc - Guido formally accepted PEP 492 with provisional status [1], so we have scope to do the following: - add abc.(Generator, Coroutine, Awaitable, AsyncIterable, AsyncIterator) in 3.5.2 (keeping the aliases in collections.abc) - drop the collections.abc aliases for the PEP 492 ABCs in 3.6 - add abc.(Callable, Iterable, Iterator) in 3.6 (keeping the aliases in collections.abc indefinitely for Python 2 compatibility) [1] https://mail.python.org/pipermail/python-dev/2015-May/139844.html
Right, the reason importlib.abc and collections.abc make sense is that when you import "importlib", you're probably interested in dynamic imports, and when you import "collections", you're probably interested in using one of the concrete container classes. The ABCs are only relevant if you're wanting to do some kind of type checking or define your own classes implementing the ABCs, so it makes sense to separate them at the module level, not just in the documentation. Other modules defining ABCs either don't need separation, or get their separation in other ways: abc: no separation needed, you're necessarily already thinking about ABCs when importing this typing: no separation needed, the only ABC is the one for defining generic types email: no separation needed, the only ABC is the one for defining email policies io: to use the io stack, you just call open() or use some other file/stream opening API numbers: to use the numeric tower, you use a builtin type, fractions.Fraction, decimal.Decimal, or some other numeric type Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Dec 31, 2015 11:48 AM, "Brett Cannon" <brett@python.org> wrote:
On Thu, Dec 31, 2015, 07:08 Alexander Walters <tritium-list@sdamon.com>
wrote:
Are these interfaces? * | Docs: http://docs.zope.org/zope.interface/ * | Docs: http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/zca.html * | Docs: https://github.com/Pylons/pyramid/blob/master/pyramid/config/zca.py * | Docs: http://docs.pylonsproject.org/projects/pyramid/en/latest/api/interfaces.html * | Src: https://github.com/Pylons/pyramid/blob/master/pyramid/interfaces.py
nothing to figured a

On Dec 31, 2015 5:56 PM, "Wes Turner" <wes.turner@gmail.com> wrote:
On Dec 31, 2015 11:48 AM, "Brett Cannon" <brett@python.org> wrote:
On Thu, Dec 31, 2015, 07:08 Alexander Walters <tritium-list@sdamon.com>
wrote:
* pyramid.interfaces.IDict http://docs.pylonsproject.org/projects/pyramid/en/latest/api/interfaces.html... * pyramid.interfaces.IMultiDict http://docs.pylonsproject.org/projects/pyramid/en/latest/api/interfaces.html... [source]
nothing to figured a them that

On 28.12.2015 18:42, Brett Cannon wrote:
I'd put them into the abc module (perhaps turning this into a package, if things get too crowded). collections.abc could then do a "from abc import *" for b/w compatibility.
-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Dec 28 2015)
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/

On 29 December 2015 at 03:58, M.-A. Lemburg <mal@egenix.com> wrote:
With the benefit of hindsight, I think a broad namespace separation like that might have been a good way to go from a data model discoverability perspective, as it clearly separates the abstract descriptions of data and control flow modelling concepts from the concrete implementations of those concepts. However, we should also keep in mind which standard library modules already publish ABCs, which is at least: typing io numbers collections.abc selectors email (That list was collected by grepping Python files for ABCMeta, so I may have missed some) That suggests to me that this design decision has effectively already been made, and it's to include the ABCs in the relevant domain specific modules. The inclusion of the non-collections related ABCs in collections.abc are the anomaly, which can be addressed by moving them out to a more appropriate location (adjusting the documentation accordingly), and then importing them into collections.abc for backwards compatibility (taking care not to increase the startup import footprint too much in the process). The current collections.abc interfaces which I think are most at issue here: Callable Iterable Iterator Generator Awaitable Coroutine Awaitable AsyncIterable AsyncIterator (I'm excluding Hashable from the list, as the main reason that matters is in describing whether or not something is suitable for inclusion in a dict or set) These differ from the rest of the collections.abc interfaces in that they're more closely associated with language level control flow syntax than they are with containers specifically: Callable - function calls Iterable - for loops, comprehensions Iterator - for loops, comprehensions Generator - generators, generator expressions Awaitable - await expressions Coroutine - async def AsyncIterable - async for AsyncIterator - async for Adding ContextManager and AsyncContextManager would give ABCs for the protocols related to "with" and "async with". Since these all correspond to syntactic protocols, I now think it's reasonable to include them directly in the "abc" namespace, since that still gives us a clear guideline for which ABCs go there, and which belong somewhere else: if it has syntax associated with it, or it's part of the ABC machinery itself, then it can go directly in the "abc" namespace. Using the "abc" namespace also ensures there isn't any additional import overhead, since that gets imported by all ABC using code anyway. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

With MAL, Nick, and Yury all suggesting the syntax-related ABCs go into the abc module and no one else really suggesting otherwise beyond me, I think it's reasonable to go with the syntax rule-of-thumb for what ABCs go into abc and all other abstract concepts go into domain-specific modules (e.g., import-related ones go into importlib.abc, which Nick forgot to list :). On Mon, 28 Dec 2015 at 19:58 Nick Coghlan <ncoghlan@gmail.com> wrote:

Would it be a good idea to mix 'concrete implementations of ABCs'* directly in the abc module where the tooling to create ABCs live, or to put it in a submodule? I feel it should be a submodule, but that isn't based on vast experience. * Yes, I know, these are not concrete implementations of types... I find it a little confusing to describe. On 12/28/2015 22:58, Nick Coghlan wrote:

On Thu, Dec 31, 2015 at 12:48 PM Brett Cannon <brett@python.org> wrote:
Locating collections ABCs in a submodule makes some sense, as there are 21 of them and the collections module is important for beginners to learn without getting distracted by ABCs. Contrast that with the direct inclusion of ABCs in most other modules and it suggests the creation of a submodule for collections may have been motivated for the same reason as this discussion -- it didn't feel right to have certain ABCs directly in the collections module. If the non-collection ABCs are being moved out of the collections module and into the ``abc`` module, there's less reason to separate them into a submodule. Beginners don't venture into the abc module expecting to understand everything. It's natural to find a bunch of ABCs in a module called ``abc``. And ABCs are included directly in many other modules instead of being relegated to a less discoverable submodule like ``typing.abc``, ``io.abc``, ``numbers.abc``, etc. as many of those are focused on ABCs in the first place. It's easy to notice a submodule when reading docs on the internet, but it's hard to figure out what the correct module is to import when hanging out at a basic REPL. Flat is better than nested and all that. * Yes, I know, these are not concrete implementations of types... I find

On 1 January 2016 at 08:18, Michael Selik <mike@selik.org> wrote:
No need to speculate, the original motive for the move is documented in the tracker: http://bugs.python.org/issue11085 (which can be found by looking at the commit history for the collections module: https://hg.python.org/cpython/log/3.5/Lib/collections/abc.py ) The problem was with folks getting confused between the abstract types like Sequence and Mapping and the usable classes like deque, ChainMap, OrderedDict, defaultdict, etc, rather than there being a lot of non-collections related content in the file. At the time, Callable was the only non-container related ABC in collections.abc - most of the others now being considered for relocation (Generator, Coroutine, Awaitable, AsyncIterable, AsyncIterator) were added as part of the PEP 492 implementation in 3.5, and *that* was mostly driven by Iterable and Iterator already being there so it was "logical" to also add Generator, AsyncIterable and AsyncIterator, with Coroutine and Awaitable coming along for the ride. That does raise the question of whether or not it's worth continuing to publish the PEP 492 ABCs from collections.abc - Guido formally accepted PEP 492 with provisional status [1], so we have scope to do the following: - add abc.(Generator, Coroutine, Awaitable, AsyncIterable, AsyncIterator) in 3.5.2 (keeping the aliases in collections.abc) - drop the collections.abc aliases for the PEP 492 ABCs in 3.6 - add abc.(Callable, Iterable, Iterator) in 3.6 (keeping the aliases in collections.abc indefinitely for Python 2 compatibility) [1] https://mail.python.org/pipermail/python-dev/2015-May/139844.html
Right, the reason importlib.abc and collections.abc make sense is that when you import "importlib", you're probably interested in dynamic imports, and when you import "collections", you're probably interested in using one of the concrete container classes. The ABCs are only relevant if you're wanting to do some kind of type checking or define your own classes implementing the ABCs, so it makes sense to separate them at the module level, not just in the documentation. Other modules defining ABCs either don't need separation, or get their separation in other ways: abc: no separation needed, you're necessarily already thinking about ABCs when importing this typing: no separation needed, the only ABC is the one for defining generic types email: no separation needed, the only ABC is the one for defining email policies io: to use the io stack, you just call open() or use some other file/stream opening API numbers: to use the numeric tower, you use a builtin type, fractions.Fraction, decimal.Decimal, or some other numeric type Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Dec 31, 2015 11:48 AM, "Brett Cannon" <brett@python.org> wrote:
On Thu, Dec 31, 2015, 07:08 Alexander Walters <tritium-list@sdamon.com>
wrote:
Are these interfaces? * | Docs: http://docs.zope.org/zope.interface/ * | Docs: http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/zca.html * | Docs: https://github.com/Pylons/pyramid/blob/master/pyramid/config/zca.py * | Docs: http://docs.pylonsproject.org/projects/pyramid/en/latest/api/interfaces.html * | Src: https://github.com/Pylons/pyramid/blob/master/pyramid/interfaces.py
nothing to figured a

On Dec 31, 2015 5:56 PM, "Wes Turner" <wes.turner@gmail.com> wrote:
On Dec 31, 2015 11:48 AM, "Brett Cannon" <brett@python.org> wrote:
On Thu, Dec 31, 2015, 07:08 Alexander Walters <tritium-list@sdamon.com>
wrote:
* pyramid.interfaces.IDict http://docs.pylonsproject.org/projects/pyramid/en/latest/api/interfaces.html... * pyramid.interfaces.IMultiDict http://docs.pylonsproject.org/projects/pyramid/en/latest/api/interfaces.html... [source]
nothing to figured a them that
participants (6)
-
Alexander Walters
-
Brett Cannon
-
M.-A. Lemburg
-
Michael Selik
-
Nick Coghlan
-
Wes Turner