PEP 520: Ordered Class Definition Namespace
I've grabbed a PEP # (520) and updated the PEP to clarify points that were brought up earlier today. Given positive feedback I got at PyCon and the reaction today, I'm hopeful the PEP isn't far off from pronouncement. :) -eric ========================================== PEP: 520 Title: Ordered Class Definition Namespace Version: $Revision$ Last-Modified: $Date$ Author: Eric Snow <ericsnowcurrently@gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 7-Jun-2016 Python-Version: 3.6 Post-History: 7-Jun-2016 Abstract ======== This PEP changes the default class definition namespace to ``OrderedDict``. Furthermore, the order in which the attributes are defined in each class body will now be preserved in ``type.__definition_order__``. This allows introspection of the original definition order, e.g. by class decorators. Note: just to be clear, this PEP is *not* about changing ``__dict__`` for classes to ``OrderedDict``. Motivation ========== Currently the namespace used during execution of a class body defaults to ``dict``. If the metaclass defines ``__prepare__()`` then the result of calling it is used. Thus, before this PEP, if you needed your class definition namespace to be ``OrderedDict`` you had to use a metaclass. Metaclasses introduce an extra level of complexity to code and in some cases (e.g. conflicts) are a problem. So reducing the need for them is worth doing when the opportunity presents itself. Given that we now have a C implementation of ``OrderedDict`` and that ``OrderedDict`` is the common use case for ``__prepare__()``, we have such an opportunity by defaulting to ``OrderedDict``. The usefulness of ``OrderedDict``-by-default is greatly increased if the definition order is directly introspectable on classes afterward, particularly by code that is independent of the original class definition. One of the original motivating use cases for this PEP is generic class decorators that make use of the definition order. Changing the default class definition namespace has been discussed a number of times, including on the mailing lists and in PEP 422 and PEP 487 (see the References section below). Specification ============= * the default class *definition* namespace is now ``OrderdDict`` * the order in which class attributes are defined is preserved in the new ``__definition_order__`` attribute on each class * "dunder" attributes (e.g. ``__init__``, ``__module__``) are ignored * ``__definition_order__`` is a tuple * ``__definition_order__`` is a read-only attribute * ``__definition_order__`` is always set: 1. if ``__definition_order__`` is defined in the class body then the value is used as-is, though the attribute will still be read-only 2. types that do not have a class definition (e.g. builtins) have their ``__definition_order__`` set to ``None`` 3. types for which `__prepare__()`` returned something other than ``OrderedDict`` (or a subclass) have their ``__definition_order__`` set to ``None`` (except where #1 applies) The following code demonstrates roughly equivalent semantics:: class Meta(type): def __prepare__(cls, *args, **kwargs): return OrderedDict() class Spam(metaclass=Meta): ham = None eggs = 5 __definition_order__ = tuple(k for k in locals() if (!k.startswith('__') or !k.endswith('__'))) Note that [pep487_] proposes a similar solution, albeit as part of a broader proposal. Why a tuple? ------------ Use of a tuple reflects the fact that we are exposing the order in which attributes on the class were *defined*. Since the definition is already complete by the time ``definition_order__`` is set, the content and order of the value won't be changing. Thus we use a type that communicates that state of immutability. Why a read-only attribute? -------------------------- As with the use of tuple, making ``__definition_order__`` a read-only attribute communicates the fact that the information it represents is complete. Since it represents the state of a particular one-time event (execution of the class definition body), allowing the value to be replaced would reduce confidence that the attribute corresponds to the original class body. If a use case for a writable (or mutable) ``__definition_order__`` arises, the restriction may be loosened later. Presently this seems unlikely and furthermore it is usually best to go immutable-by-default. Note that ``__definition_order__`` is centered on the class definition body. The use cases for dealing with the class namespace (``__dict__``) post-definition are a separate matter. ``__definition_order__`` would be a significantly misleading name for a supporting feature. See [nick_concern_] for more discussion. Why ignore "dunder" names? -------------------------- Names starting and ending with "__" are reserved for use by the interpreter. In practice they should not be relevant to the users of ``__definition_order__``. Instead, for early everyone they would only be clutter, causing the same extra work for everyone. Why is __definition_order__ even necessary? ------------------------------------------- Since the definition order is not preserved in ``__dict__``, it would be lost once class definition execution completes. Classes *could* explicitly set the attribute as the last thing in the body. However, then independent decorators could only make use of classes that had done so. Instead, ``__definition_order__`` preserves this one bit of info from the class body so that it is universally available. Compatibility ============= This PEP does not break backward compatibility, except in the case that someone relies *strictly* on ``dict`` as the class definition namespace. This shouldn't be a problem. Changes ============= In addition to the class syntax, the following expose the new behavior: * builtins.__build_class__ * types.prepare_class * types.new_class Other Python Implementations ============================ Pending feedback, the impact on Python implementations is expected to be minimal. If a Python implementation cannot support switching to `OrderedDict``-by-default then it can always set ``__definition_order__`` to ``None``. Implementation ============== The implementation is found in the tracker. [impl_] Alternatives ============ type.__dict__ as OrderedDict ---------------------------- Instead of storing the definition order in ``__definition_order__``, the now-ordered definition namespace could be copied into a new ``OrderedDict``. This would mostly provide the same semantics. However, using ``OrderedDict`` for ``type,__dict__`` would obscure the relationship with the definition namespace, making it less useful. Additionally, doing this would require significant changes to the semantics of the concrete ``dict`` C-API. A "namespace" Keyword Arg for Class Definition ---------------------------------------------- PEP 422 introduced a new "namespace" keyword arg to class definitions that effectively replaces the need to ``__prepare__()``. [pep422_] However, the proposal was withdrawn in favor of the simpler PEP 487. References ========== .. [impl] issue #24254 (https://bugs.python.org/issue24254) .. [nick_concern] Nick's concerns about mutability (https://mail.python.org/pipermail/python-dev/2016-June/144883.html) .. [pep422] PEP 422 (https://www.python.org/dev/peps/pep-0422/#order-preserving-classes) .. [pep487] PEP 487 (https://www.python.org/dev/peps/pep-0487/#defining-arbitrary-namespaces) .. [orig] original discussion (https://mail.python.org/pipermail/python-ideas/2013-February/019690.html) .. [followup1] follow-up 1 (https://mail.python.org/pipermail/python-dev/2013-June/127103.html) .. [followup2] follow-up 2 (https://mail.python.org/pipermail/python-dev/2015-May/140137.html) Copyright =========== This document has been placed in the public domain.
On 06/07/2016 05:50 PM, Eric Snow wrote: Overall +1. Some nits below.
Specification =============
3. types for which `__prepare__()`` returned something other than ``OrderedDict`` (or a subclass) have their ``__definition_order__`` set to ``None``
(unless ``__definition_order__`` is present in the class dict either by virtue of being in the class body or because the metaclass inserted it before calling ``type.__new__``)
__definition_order__ = tuple(k for k in locals() if (!k.startswith('__') or !k.endswith('__')))
Still mixing C and Python! ;)
Why a tuple? ------------
Use of a tuple reflects the fact that we are exposing the order in which attributes on the class were *defined*. Since the definition is already complete by the time ``definition_order__`` is set, the content and order of the value won't be changing. Thus we use a type that communicates that state of immutability.
Why a read-only attribute? --------------------------
As with the use of tuple, making ``__definition_order__`` a read-only attribute communicates the fact that the information it represents is complete. Since it represents the state of a particular one-time event (execution of the class definition body), allowing the value to be replaced would reduce confidence that the attribute corresponds to the original class body.
If a use case for a writable (or mutable) ``__definition_order__`` arises, the restriction may be loosened later. Presently this seems unlikely and furthermore it is usually best to go immutable-by-default.
If __definition_order__ is supposed to be immutable as well as read-only then we should convert non-tuples to tuples. No point in letting that user bug slip through.
Why ignore "dunder" names? --------------------------
Names starting and ending with "__" are reserved for use by the interpreter. In practice they should not be relevant to the users of ``__definition_order__``. Instead, for early everyone they would only
s/early/nearly
Why is __definition_order__ even necessary? -------------------------------------------
Since the definition order is not preserved in ``__dict__``, it would be lost once class definition execution completes. Classes *could* explicitly set the attribute as the last thing in the body. However, then independent decorators could only make use of classes that had done so. Instead, ``__definition_order__`` preserves this one bit of info from the class body so that it is universally available.
s/would be/is -- ~Ethan~
On Tue, Jun 7, 2016 at 6:20 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
On 06/07/2016 05:50 PM, Eric Snow wrote:
__definition_order__ = tuple(k for k in locals() if (!k.startswith('__') or !k.endswith('__')))
Still mixing C and Python! ;)
I knew I was missing something!
Why a tuple? ------------
Use of a tuple reflects the fact that we are exposing the order in which attributes on the class were *defined*. Since the definition is already complete by the time ``definition_order__`` is set, the content and order of the value won't be changing. Thus we use a type that communicates that state of immutability.
Why a read-only attribute? --------------------------
As with the use of tuple, making ``__definition_order__`` a read-only attribute communicates the fact that the information it represents is complete. Since it represents the state of a particular one-time event (execution of the class definition body), allowing the value to be replaced would reduce confidence that the attribute corresponds to the original class body.
If a use case for a writable (or mutable) ``__definition_order__`` arises, the restriction may be loosened later. Presently this seems unlikely and furthermore it is usually best to go immutable-by-default.
If __definition_order__ is supposed to be immutable as well as read-only then we should convert non-tuples to tuples. No point in letting that user bug slip through.
Do you mean if a class explicitly defines __definition_order__? If so, I'm not clear on how that would work. It could be set to anything, including None or a value that does not iterate into a definition order. If someone explicitly set __definition_order__ then I think it should be used as-is.
Why ignore "dunder" names? --------------------------
Names starting and ending with "__" are reserved for use by the interpreter. In practice they should not be relevant to the users of ``__definition_order__``. Instead, for early everyone they would only
s/early/nearly
fixed
Why is __definition_order__ even necessary? -------------------------------------------
Since the definition order is not preserved in ``__dict__``, it would be lost once class definition execution completes. Classes *could* explicitly set the attribute as the last thing in the body. However, then independent decorators could only make use of classes that had done so. Instead, ``__definition_order__`` preserves this one bit of info from the class body so that it is universally available.
s/would be/is
fixed Thanks! -eric
On 7 June 2016 at 20:17, Eric Snow <ericsnowcurrently@gmail.com> wrote:
On Tue, Jun 7, 2016 at 6:20 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
If __definition_order__ is supposed to be immutable as well as read-only then we should convert non-tuples to tuples. No point in letting that user bug slip through.
Do you mean if a class explicitly defines __definition_order__? If so, I'm not clear on how that would work. It could be set to anything, including None or a value that does not iterate into a definition order. If someone explicitly set __definition_order__ then I think it should be used as-is.
I'm guessing Ethan is suggesting defining it as: __definition_order__ = tuple(ns["__definition_order__"]) When the attribute is present in the method body. That restriction would be comparable to what we do with __slots__ today: >>> class C: ... __slots__ = 1 ... Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'int' object is not iterable Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 06/09/2016 02:39 PM, Nick Coghlan wrote:
On 7 June 2016 at 20:17, Eric Snow wrote:
On Tue, Jun 7, 2016 at 6:20 PM, Ethan Furman wrote:
If __definition_order__ is supposed to be immutable as well as read-only then we should convert non-tuples to tuples. No point in letting that user bug slip through.
Do you mean if a class explicitly defines __definition_order__? If so, I'm not clear on how that would work. It could be set to anything, including None or a value that does not iterate into a definition order. If someone explicitly set __definition_order__ then I think it should be used as-is.
I'm guessing Ethan is suggesting defining it as:
__definition_order__ = tuple(ns["__definition_order__"])
When the attribute is present in the method body.
Yup, that it's it exactly. Thanks, Nick! -- ~Ethan~
On Thu, Jun 9, 2016 at 2:39 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I'm guessing Ethan is suggesting defining it as:
__definition_order__ = tuple(ns["__definition_order__"])
When the attribute is present in the method body.
Ah. I'd rather stick to "consenting adults" in the case that __definition_order__ is explicitly set. We'll strongly recommend setting it to None or a tuple of identifier strings.
That restriction would be comparable to what we do with __slots__ today:
>>> class C: ... __slots__ = 1 ... Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'int' object is not iterable
Are you suggesting that we require it be a tuple of identifiers (or None) and raise TypeError otherwise, similar to __slots__? The difference is that __slots__ has specific type requirements that do not apply to __definition_order__, as well as a different purpose. __definition_order__ is about preserving definition-type info that we are currently throwing away. -eric
On 10 June 2016 at 09:42, Eric Snow <ericsnowcurrently@gmail.com> wrote:
On Thu, Jun 9, 2016 at 2:39 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
That restriction would be comparable to what we do with __slots__ today:
>>> class C: ... __slots__ = 1 ... Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'int' object is not iterable
Are you suggesting that we require it be a tuple of identifiers (or None) and raise TypeError otherwise, similar to __slots__? The difference is that __slots__ has specific type requirements that do not apply to __definition_order__, as well as a different purpose. __definition_order__ is about preserving definition-type info that we are currently throwing away.
If we don't enforce the tuple-of-identifiers restriction at type creation time, everyone that *doesn't* make it a tuple-of-identifiers is likely to have a subtle compatibility bug with class decorators and other code that assume the default tuple-of-identifiers format is the only possible format (aside from None). To put it in PEP 484 terms: regardless of what the PEP says, people are going to assume the type of __definition_order__ is Optional[Tuple[str]], as that's going to cover almost all class definitions they encounter. It makes sense to me to give class definitions and metaclasses the opportunity to change the *content* of the definition order: "Use these names in this order, not the names and order you would have calculated by default". It doesn't make sense to me to give them an opportunity to change the *form* of the definition order, since that makes it incredibly difficult to consume correctly: "Sure, it's *normally* a tuple-of-identifiers, but it *might* be a dictionary, or a complex number, or a set, or whatever the class author decided to make it". By contrast, if the class machinery enforces Optional[Tuple[str]], then it becomes a lot easier to consume reliably, and anyone violating the constraint gets an immediate exception when defining the offending class, rather than a potentially obscure exception from a class decorator or other piece of code that assumes __definition_order__ could only be None or a tuple of strings. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, Jun 10, 2016 at 11:29 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 10 June 2016 at 09:42, Eric Snow <ericsnowcurrently@gmail.com> wrote:
On Thu, Jun 9, 2016 at 2:39 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
That restriction would be comparable to what we do with __slots__ today:
>>> class C: ... __slots__ = 1 ... Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'int' object is not iterable
Are you suggesting that we require it be a tuple of identifiers (or None) and raise TypeError otherwise, similar to __slots__? The difference is that __slots__ has specific type requirements that do not apply to __definition_order__, as well as a different purpose. __definition_order__ is about preserving definition-type info that we are currently throwing away.
If we don't enforce the tuple-of-identifiers restriction at type creation time, everyone that *doesn't* make it a tuple-of-identifiers is likely to have a subtle compatibility bug with class decorators and other code that assume the default tuple-of-identifiers format is the only possible format (aside from None). To put it in PEP 484 terms: regardless of what the PEP says, people are going to assume the type of __definition_order__ is Optional[Tuple[str]], as that's going to cover almost all class definitions they encounter.
It makes sense to me to give class definitions and metaclasses the opportunity to change the *content* of the definition order: "Use these names in this order, not the names and order you would have calculated by default".
It doesn't make sense to me to give them an opportunity to change the *form* of the definition order, since that makes it incredibly difficult to consume correctly: "Sure, it's *normally* a tuple-of-identifiers, but it *might* be a dictionary, or a complex number, or a set, or whatever the class author decided to make it".
By contrast, if the class machinery enforces Optional[Tuple[str]], then it becomes a lot easier to consume reliably, and anyone violating the constraint gets an immediate exception when defining the offending class, rather than a potentially obscure exception from a class decorator or other piece of code that assumes __definition_order__ could only be None or a tuple of strings.
That makes sense. I'll adjust the PEP (and the implementation). -eric
On Jun 7, 2016 8:52 PM, "Eric Snow" <ericsnowcurrently@gmail.com> wrote:
* the default class *definition* namespace is now ``OrderdDict`` * the order in which class attributes are defined is preserved in the
By using an OrderedDict, names are ordered by first definition point, rather than location of the used definition. For example, the definition order of the following will be "x, y", even though the definitions actually bound to the name are in order "y, x". class C: x = 0 def y(self): return 'y' def x(self): return 'x' Is that okay?
On Wed, Jun 8, 2016 at 12:07 AM, Franklin? Lee <leewangzhong+python@gmail.com> wrote:
On Jun 7, 2016 8:52 PM, "Eric Snow" <ericsnowcurrently@gmail.com> wrote:
* the default class *definition* namespace is now ``OrderdDict`` * the order in which class attributes are defined is preserved in the
By using an OrderedDict, names are ordered by first definition point, rather than location of the used definition.
For example, the definition order of the following will be "x, y", even though the definitions actually bound to the name are in order "y, x". class C: x = 0 def y(self): return 'y' def x(self): return 'x'
Is that okay?
In practice that will seldom be an issue. In the few cases where it could possibly be a problem, the class may explicitly set __definition_order__. -eric
Abstract ========
This PEP changes the default class definition namespace to ``OrderedDict``. Furthermore, the order in which the attributes are defined in each class body will now be preserved in ``type.__definition_order__``. This allows introspection of the original definition order, e.g. by class decorators.
Note: just to be clear, this PEP is *not* about changing ``__dict__`` for classes to ``OrderedDict``.
What is the cost in term of performance? What can be slower: define a new class and/or instanciate a class? Victor
On Wed, Jun 8, 2016 at 1:07 AM, Victor Stinner <victor.stinner@gmail.com> wrote:
Abstract ========
This PEP changes the default class definition namespace to ``OrderedDict``. Furthermore, the order in which the attributes are defined in each class body will now be preserved in ``type.__definition_order__``. This allows introspection of the original definition order, e.g. by class decorators.
Note: just to be clear, this PEP is *not* about changing ``__dict__`` for classes to ``OrderedDict``.
What is the cost in term of performance?
Do you mean the cost of the PEP? The extra cost is negligible: creating an OrderedDict + mutation operations on it. Note that it is only used during class definition (execution of the class body).
What can be slower: define a new class and/or instanciate a class?
By "instantiate" do you mean the equivalent of "type(...)" or do you mean creating a new instance of a class? As noted above, the impact of using OrderedDict during class definition is negligible. During definition the cost of other operations will usually dwarf any extra overhead from using an OrderedDict. -eric
On 6/8/2016 4:07 AM, Victor Stinner wrote:
Abstract ========
This PEP changes the default class definition namespace to ``OrderedDict``. Furthermore, the order in which the attributes are defined in each class body will now be preserved in ``type.__definition_order__``. This allows introspection of the original definition order, e.g. by class decorators.
Note: just to be clear, this PEP is *not* about changing ``__dict__`` for classes to ``OrderedDict``.
What is the cost in term of performance?
What can be slower: define a new class and/or instanciate a class?
A class is defined once, used many times to instantiate instances. Each instance is typically used many times, with many lookups. So it is self.class_attribute lookups, like method lookups, that likely matter the most, and which are not changed by the PEP. -- Terry Jan Reedy
Is there any rationale for rejecting alternatives like: 1. Adding standard metaclass with ordered namespace. 2. Adding `namespace` or `ordered` args to the default metaclass. 3. Making compiler fill in __definition_order__ for every class (just like __qualname__) without touching the runtime. ? To me, any of the above seems preferred to complicating the core part of the language forever. The vast majority of Python classes don't care about their member order, this is minority use case receiving majority treatment. Also, wiring OrderedDict into class creation means elevating it from a peripheral utility to indispensable built-in type.
On 14 June 2016 at 02:41, Nikita Nemkin <nikita@nemkin.ru> wrote:
Is there any rationale for rejecting alternatives like:
Good questions - Eric, it's likely worth capturing answers to these in the PEP for the benefit of future readers.
1. Adding standard metaclass with ordered namespace.
Adding metaclasses to an existing class can break compatibility with third party subclasses, so making it possible for people to avoid that while still gaining the ability to implicitly expose attribute ordering to class decorators and other potentially interested parties is a recurring theme behind this PEP and also PEPs 422 and 487.
2. Adding `namespace` or `ordered` args to the default metaclass.
See below (as it relates to your own complexity argument)
3. Making compiler fill in __definition_order__ for every class (just like __qualname__) without touching the runtime. ?
Class scopes support conditionals and loops, so we can't necessarily be sure what names will be assigned without running the code. It's also possible to make attribute assignments via locals() that are entirely opaque to the compiler, but visible to the interpreter at runtime.
To me, any of the above seems preferred to complicating the core part of the language forever.
The vast majority of Python classes don't care about their member order, this is minority use case receiving majority treatment.
Also, wiring OrderedDict into class creation means elevating it from a peripheral utility to indispensable built-in type.
Right, that's one of the key reasons this is a PEP, rather than just an item on the issue tracker. The rationale for "Why not make this configurable, rather than switching it unilaterally?" is that it's actually *simpler* overall to just make it the default - we can then change the documentation to say "class bodies are evaluated in a collections.OrderedDict instance by default" and record the consequences of that, rather than having to document yet another class customisation mechanism. It also eliminates boilerplate from class decorator usage instructions, where people have to write "to use this class decorator, you must also specify 'namespace=collections.OrderedDict' in your class header" Folks that don't need the ordering information do end up paying a slight import time and memory cost, which is another key reason for handling the proposal as a PEP rather than just as a tracker issue. Aside from the boilerplate reduction when used in conjunction with a class decorator, a further possible category of consumers would be documentation generators like pydoc and Sphinx apidoc, which may be able to switch to displaying methods in definition order, rather than the current approach of always listing them in alphabetical order. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Thanks for raising these good points, Nikita. I'll make sure the PEP reflects this discussion. (inline responses below...) -eric On Tue, Jun 14, 2016 at 3:41 AM, Nikita Nemkin <nikita@nemkin.ru> wrote:
Is there any rationale for rejecting alternatives like:
1. Adding standard metaclass with ordered namespace. 2. Adding `namespace` or `ordered` args to the default metaclass.
We already have a metaclass-based solution: __prepare__(). Unfortunately, this opt-in option means that the definition order isn't preserved by default, which means folks can't rely on access to the definition order. This is effectively no different from the status quo. Furthermore, there's a practical problem with requiring the use of metaclasses to achieve some particular capability: metaclass conflicts. PEPs 422 and 487 exist, in large part, as a response to specific feedback from users about problems they've had with metaclasses. While the key objective of PEP 520 is preserving the class definition order, it also helps make it less necessary to write a metaclass.
3. Making compiler fill in __definition_order__ for every class (just like __qualname__) without touching the runtime.
This is a great idea. I'd support any effort to do so. But keep in mind that how we derive __definition_order__ isn't as important as that it's always there. So the use of OrderedDict for the implementation isn't necessary. Instead, it's the implementation I've taken. If we later switch to using the compiler to get the definition order, then great!
?
To me, any of the above seems preferred to complicating the core part of the language forever.
What specific complication are you expecting? Like nearly all of Python's "power tools", folks won't need to know about the changes from this PEP in order to use the language. Then when they need the new functionality, it will be ready for them to use. Furthermore, as far as changes to the language go, this change is quite simple and straightforward (consider other recent changes, e.g. async). It is arguably a natural step and fills in some of the information that Python currently throws away. Finally, I've gotten broad support for the change from across the community (both on the mailing lists and in personal correspondence), from the time I first introduced the idea several years ago.
The vast majority of Python classes don't care about their member order, this is minority use case receiving majority treatment.
The problem is that there isn't any other recourse available to code that wishes to determine the definition order of an arbitrary class. This is an obstacle to code that I personally want to write (hence my interest).
Also, wiring OrderedDict into class creation means elevating it from a peripheral utility to indispensable built-in type.
Note that as of 3.5 CPython's OrderedDict *is* a builtin type (though exposed via the collections module rather than the builtins module). However, you're right that this change would mean OrderedDict would now be used by the interpreter in all implementations of Python 3.6+. Some of the other implementators from which I've gotten feedback have indicated this isn't a problem.
On 7 June 2016 at 17:50, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Why is __definition_order__ even necessary? -------------------------------------------
Since the definition order is not preserved in ``__dict__``, it would be lost once class definition execution completes. Classes *could* explicitly set the attribute as the last thing in the body. However, then independent decorators could only make use of classes that had done so. Instead, ``__definition_order__`` preserves this one bit of info from the class body so that it is universally available.
The discussion in the PEP 487 thread made me realise that I'd like to see a discussion in PEP 520 regarding whether or not to define __definition_order__ for builtin types initialised via PyType_Ready or created via PyType_FromSpec in addition to defining it for types created via the class statement or types.new_class(). For static types, PyType_Ready could potentially set it based on tp_members, tp_methods & tp_getset (see https://docs.python.org/3/c-api/typeobj.html ) Similarly, PyType_FromSpec could potentially set it based on the contents of Py_tp_members, Py_tp_methods and Py_tp_getset slot definitions Having definition order support in both types.new_class() and builtin types would also make it clear why we can't rely purely on the compiler to provide the necessary ordering information - in both of those cases, the Python compiler isn't directly involved in the type creation process. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I agree it's better to define the order as computed at runtime. I don't think there's much of a point to mandate that all builtin/extension types reveal their order too -- I doubt there will be many uses for that -- but I don't want to disallow it either. But we can allow types to define this, as long as it's in their documentation (so users can rely on it in those cases). As another point of review, I don't like the exception for dunder names. I can see that __module__, __name__ etc. are distractions, but since you're adding methods, you should also add methods with dunder names. The overlap with PEP 487 makes me think that this feature is clearly desirable (I like the name you give it in PEP 520 better, and PEP 487 is too vague about its definition). Finally, it seems someone is working on making all dicts ordered. Does that mean this will soon be obsolete? On Fri, Jun 17, 2016 at 6:32 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 7 June 2016 at 17:50, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Why is __definition_order__ even necessary? -------------------------------------------
Since the definition order is not preserved in ``__dict__``, it would be lost once class definition execution completes. Classes *could* explicitly set the attribute as the last thing in the body. However, then independent decorators could only make use of classes that had done so. Instead, ``__definition_order__`` preserves this one bit of info from the class body so that it is universally available.
The discussion in the PEP 487 thread made me realise that I'd like to see a discussion in PEP 520 regarding whether or not to define __definition_order__ for builtin types initialised via PyType_Ready or created via PyType_FromSpec in addition to defining it for types created via the class statement or types.new_class().
For static types, PyType_Ready could potentially set it based on tp_members, tp_methods & tp_getset (see https://docs.python.org/3/c-api/typeobj.html ) Similarly, PyType_FromSpec could potentially set it based on the contents of Py_tp_members, Py_tp_methods and Py_tp_getset slot definitions
Having definition order support in both types.new_class() and builtin types would also make it clear why we can't rely purely on the compiler to provide the necessary ordering information - in both of those cases, the Python compiler isn't directly involved in the type creation process.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On Mon, Jun 20, 2016 at 9:49 AM, Guido van Rossum <guido@python.org> wrote:
I agree it's better to define the order as computed at runtime. I don't think there's much of a point to mandate that all builtin/extension types reveal their order too -- I doubt there will be many uses for that -- but I don't want to disallow it either. But we can allow types to define this, as long as it's in their documentation (so users can rely on it in those cases).
Agreed.
As another point of review, I don't like the exception for dunder names. I can see that __module__, __name__ etc. are distractions, but since you're adding methods, you should also add methods with dunder names.
I still think that in practice the dunder names will be clutter that folks have to ignore. However, it's a relatively weak point given that it's easy to ignore dunder names. So I don't mind including them.
The overlap with PEP 487 makes me think that this feature is clearly desirable (I like the name you give it in PEP 520 better, and PEP 487 is too vague about its definition).
Agreed.
Finally, it seems someone is working on making all dicts ordered. Does that mean this will soon be obsolete?
Nope. Having an ordered definition namespace by default does not give us __definition_order__ for free. Furthermore, the compact dict under consideration isn't strictly order-preserving (re-orders for deletion). -eric
Finally, it seems someone is working on making all dicts ordered. Does that mean this will soon be obsolete?
Nope. Having an ordered definition namespace by default does not give us __definition_order__ for free. Furthermore, the compact dict under consideration isn't strictly order-preserving (re-orders for deletion).
compact ordered dict I proposed is preserves insertion order even some items are deleted. http://bugs.python.org/issue27350 Should I post PEP for compact dict? Here is my draft, but I haven't posted it yet since my English is much worse than C. https://www.dropbox.com/s/s85n9b2309k03cq/pep-compact-dict.txt?dl=0
Hi! On Tue, Jun 21, 2016 at 11:14:39AM +0900, INADA Naoki <songofacandy@gmail.com> wrote:
Here is my draft, but I haven't posted it yet since my English is much worse than C. https://www.dropbox.com/s/s85n9b2309k03cq/pep-compact-dict.txt?dl=0
It's good enough for a start (if a PEP is needed at all). If you push it to Github I'm sure they will come with pull requests. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Tue, Jun 21, 2016 at 12:17 PM, Oleg Broytman <phd@phdru.name> wrote:
Hi!
On Tue, Jun 21, 2016 at 11:14:39AM +0900, INADA Naoki <songofacandy@gmail.com> wrote:
Here is my draft, but I haven't posted it yet since my English is much worse than C. https://www.dropbox.com/s/s85n9b2309k03cq/pep-compact-dict.txt?dl=0
It's good enough for a start (if a PEP is needed at all). If you push it to Github I'm sure they will come with pull requests.
Oleg.
Thank you for reading my draft.
(if a PEP is needed at all)
I don't think so. My PEP is not for changing Python Language, just describe implementation detail. Python 3.5 has new OrderedDict implemented in C without PEP. My patch is relatively small than it. And the idea has been well known. -- INADA Naoki <songofacandy@gmail.com>
On Mon, Jun 20, 2016 at 11:02 PM, INADA Naoki <songofacandy@gmail.com> wrote:
On Tue, Jun 21, 2016 at 12:17 PM, Oleg Broytman <phd@phdru.name> wrote:
(if a PEP is needed at all)
I don't think so. My PEP is not for changing Python Language, just describe implementation detail.
Python 3.5 has new OrderedDict implemented in C without PEP. My patch is relatively small than it. And the idea has been well known.
How about, for 3.6, target re-implementing OrderedDict using the compact dict approach (and leave dict alone for now). That way we have an extra release cycle to iron out the kinks before switching dict over for 3.7. :) -eric
On Fri, Jun 24, 2016 at 12:03 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
On Mon, Jun 20, 2016 at 11:02 PM, INADA Naoki <songofacandy@gmail.com> wrote:
On Tue, Jun 21, 2016 at 12:17 PM, Oleg Broytman <phd@phdru.name> wrote:
(if a PEP is needed at all)
I don't think so. My PEP is not for changing Python Language, just describe implementation detail.
Python 3.5 has new OrderedDict implemented in C without PEP. My patch is relatively small than it. And the idea has been well known.
How about, for 3.6, target re-implementing OrderedDict using the compact dict approach (and leave dict alone for now). That way we have an extra release cycle to iron out the kinks before switching dict over for 3.7. :)
-eric
I can't. Since OrderedDict inherits dict. OrderedDict implementation based on dict implementation. Since I'm not expert of Python object system, I don't know how to separate OrderedDict implementation from dict. -- INADA Naoki <songofacandy@gmail.com>
There are a number of ways to make it work (mostly). However, I'll defer to Raymond on how strictly OrderedDict should "subclass" from dict. -eric On Thu, Jun 23, 2016 at 9:26 AM, INADA Naoki <songofacandy@gmail.com> wrote:
On Fri, Jun 24, 2016 at 12:03 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
On Mon, Jun 20, 2016 at 11:02 PM, INADA Naoki <songofacandy@gmail.com> wrote:
On Tue, Jun 21, 2016 at 12:17 PM, Oleg Broytman <phd@phdru.name> wrote:
(if a PEP is needed at all)
I don't think so. My PEP is not for changing Python Language, just describe implementation detail.
Python 3.5 has new OrderedDict implemented in C without PEP. My patch is relatively small than it. And the idea has been well known.
How about, for 3.6, target re-implementing OrderedDict using the compact dict approach (and leave dict alone for now). That way we have an extra release cycle to iron out the kinks before switching dict over for 3.7. :)
-eric
I can't. Since OrderedDict inherits dict. OrderedDict implementation based on dict implementation. Since I'm not expert of Python object system, I don't know how to separate OrderedDict implementation from dict.
-- INADA Naoki <songofacandy@gmail.com>
FYI, Here is calculated size of each dict by len(d). https://docs.google.com/spreadsheets/d/1nN5y6IsiJGdNxD7L7KBXmhdUyXjuRAQR_Wbr... On Tue, Jun 21, 2016 at 12:17 PM, Oleg Broytman <phd@phdru.name> wrote:
Hi!
On Tue, Jun 21, 2016 at 11:14:39AM +0900, INADA Naoki <songofacandy@gmail.com> wrote:
Here is my draft, but I haven't posted it yet since my English is much worse than C. https://www.dropbox.com/s/s85n9b2309k03cq/pep-compact-dict.txt?dl=0
It's good enough for a start (if a PEP is needed at all). If you push it to Github I'm sure they will come with pull requests.
Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com
-- INADA Naoki <songofacandy@gmail.com>
On Fri, Jun 17, 2016 at 7:32 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
The discussion in the PEP 487 thread made me realise that I'd like to see a discussion in PEP 520 regarding whether or not to define __definition_order__ for builtin types initialised via PyType_Ready or created via PyType_FromSpec in addition to defining it for types created via the class statement or types.new_class().
For static types, PyType_Ready could potentially set it based on tp_members, tp_methods & tp_getset (see https://docs.python.org/3/c-api/typeobj.html ) Similarly, PyType_FromSpec could potentially set it based on the contents of Py_tp_members, Py_tp_methods and Py_tp_getset slot definitions
Having definition order support in both types.new_class() and builtin types would also make it clear why we can't rely purely on the compiler to provide the necessary ordering information - in both of those cases, the Python compiler isn't directly involved in the type creation process.
I'll mention this in the PEP, but I'd rather not make it a part of the proposal. -eric
participants (10)
-
Eric Snow -
Ethan Furman -
Franklin? Lee -
Guido van Rossum -
INADA Naoki -
Nick Coghlan -
Nikita Nemkin -
Oleg Broytman -
Terry Reedy -
Victor Stinner