How to update namedtuple asdict() to use dict instead of OrderedDict
Now that regular dicts are ordered and compact, it makes more sense for the _asdict() method to create a regular dict (as it did in its early days) rather than an OrderedDict. The regular dict is much smaller, much faster, and has a much cleaner looking repr. It would also help namedtuple() stay in sync with dataclasses which already take advantage of the ordering feature of regular dicts. The question is how to be go about making the change in a way gives the most benefit to users as soon as possible and that creates the least disruption. Option 1) Add a deprecation notice to 3.8, make no code change in 3.8, and then update the code in 3.9. This has several issues: a) it doesn't provide an executable DeprecationWarning in 3.8, b) it isn't really a deprecation, and c) it defers the benefits of the change for another release. Option 2) Add a deprecation notice to 3.8, add a DeprecationWarning to the _asdict() method, and make the actual improvement in 3.9. The main issue here is that it will create a lot of noise for normal uses of the _asdict() method which are otherwise unaffected by the change. The typical use cases for _asdict() are to create keyword arguments and to pass named tuple data into functions or methods that expect regular dictionaries. Those use cases would benefit from seeing the change made sooner and would suffer in the interim from their code slowing down for warnings that aren't useful. Option 3). Add a deprecation notice to 3.8 and have the _asdict() method create a subclass of OrderedDict that issues warnings only for the methods and attributes that will change (move_to_end, popitem, __eq__, __dict__, __weakref__). This is less noisy but it adds a lot of machinery just to make a notification of a minor change. Also, it fails to warn that the data type will change. And it may create more confusion than options 1 and 4 which are simpler. Option 4) Just make the change directly in 3.8, s/OrderedDict/dict/, and be done will it. This gives users the benefits right away and doesn't annoy them with warnings that they likely don't care about. There is some precedent for this. To make namedtuple class creation faster, the *verbose* option was dropped without any deprecation period. It looks like no one missed that feature at all, but they did get the immediate benefit of faster import times. In the case of using regular dicts in named tuples, people will get immediate and significant space savings as well as a speed benefit. My recommendation is Option 4 as being less disruptive and more beneficial than the other options. In the unlikely event that anyone is currently depending on the reordering methods for the output of _asdict(), the remediation is trivially simple: nt._asdict() -> OrderedDict(nt.as_dict()). What do you all think? Raymond
On Wed, 30 Jan 2019 at 22:35, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
My recommendation is Option 4 as being less disruptive and more beneficial than the other options. In the unlikely event that anyone is currently depending on the reordering methods for the output of _asdict(), the remediation is trivially simple: nt._asdict() -> OrderedDict(nt.as_dict()).
What do you all think?
+1 from me on option 4. Paul
On 01/30/2019 02:55 PM, Paul Moore wrote:
On Wed, 30 Jan 2019 at 22:35, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
My recommendation is Option 4 as being less disruptive and more beneficial than the other options. In the unlikely event that anyone is currently depending on the reordering methods for the output of _asdict(), the remediation is trivially simple: nt._asdict() -> OrderedDict(nt.as_dict()).
What do you all think?
+1 from me on option 4.
From me as well. -- ~Ethan~
Ditto +1 option 4 On Wed, Jan 30, 2019, 5:56 PM Paul Moore <p.f.moore@gmail.com wrote:
My recommendation is Option 4 as being less disruptive and more beneficial than the other options. In the unlikely event that anyone is currently depending on the reordering methods for the output of _asdict(),
On Wed, 30 Jan 2019 at 22:35, Raymond Hettinger <raymond.hettinger@gmail.com> wrote: the remediation is trivially simple: nt._asdict() -> OrderedDict(nt.as_dict()).
What do you all think?
+1 from me on option 4.
Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/mertz%40gnosis.cx
On 1/30/2019 2:32 PM, Raymond Hettinger wrote:
Now that regular dicts are ordered and compact, it makes more sense for the _asdict() method to create a regular dict (as it did in its early days) rather than an OrderedDict. ... Option 4) Just make the change directly in 3.8, s/OrderedDict/dict/, and be done will it. This gives users the benefits right away and doesn't annoy them with warnings that they likely don't care about. There is some precedent for this. To make namedtuple class creation faster, the *verbose* option was dropped without any deprecation period. It looks like no one missed that feature at all, but they did get the immediate benefit of faster import times. In the case of using regular dicts in named tuples, people will get immediate and significant space savings as well as a speed benefit.
My recommendation is Option 4 as being less disruptive and more beneficial than the other options. In the unlikely event that anyone is currently depending on the reordering methods for the output of _asdict(), the remediation is trivially simple: nt._asdict() -> OrderedDict(nt.as_dict()).
What do you all think? Option 4 sounds good to me.
Would it be practical to add deprecated methods to regular dict for the OrderedDict reordering methods that raise with an error suggesting "To use this method, convert dict to OrderedDict." (or some better wording).
On Wed, Jan 30, 2019 at 2:32 PM Raymond Hettinger < raymond.hettinger@gmail.com> wrote:
Now that regular dicts are ordered and compact, it makes more sense for the _asdict() method to create a regular dict (as it did in its early days) rather than an OrderedDict. The regular dict is much smaller, much faster, and has a much cleaner looking repr. It would also help namedtuple() stay in sync with dataclasses which already take advantage of the ordering feature of regular dicts.
The question is how to be go about making the change in a way gives the most benefit to users as soon as possible and that creates the least disruption.
Option 1) Add a deprecation notice to 3.8, make no code change in 3.8, and then update the code in 3.9. This has several issues: a) it doesn't provide an executable DeprecationWarning in 3.8, b) it isn't really a deprecation, and c) it defers the benefits of the change for another release.
Option 2) Add a deprecation notice to 3.8, add a DeprecationWarning to the _asdict() method, and make the actual improvement in 3.9. The main issue here is that it will create a lot of noise for normal uses of the _asdict() method which are otherwise unaffected by the change. The typical use cases for _asdict() are to create keyword arguments and to pass named tuple data into functions or methods that expect regular dictionaries. Those use cases would benefit from seeing the change made sooner and would suffer in the interim from their code slowing down for warnings that aren't useful.
Option 3). Add a deprecation notice to 3.8 and have the _asdict() method create a subclass of OrderedDict that issues warnings only for the methods and attributes that will change (move_to_end, popitem, __eq__, __dict__, __weakref__). This is less noisy but it adds a lot of machinery just to make a notification of a minor change. Also, it fails to warn that the data type will change. And it may create more confusion than options 1 and 4 which are simpler.
Option 4) Just make the change directly in 3.8, s/OrderedDict/dict/, and be done will it. This gives users the benefits right away and doesn't annoy them with warnings that they likely don't care about. There is some precedent for this. To make namedtuple class creation faster, the *verbose* option was dropped without any deprecation period. It looks like no one missed that feature at all, but they did get the immediate benefit of faster import times. In the case of using regular dicts in named tuples, people will get immediate and significant space savings as well as a speed benefit.
My recommendation is Option 4 as being less disruptive and more beneficial than the other options. In the unlikely event that anyone is currently depending on the reordering methods for the output of _asdict(), the remediation is trivially simple: nt._asdict() -> OrderedDict(nt.as_dict()).
+1 on option 4. I agree with everyone else. Because the remediation that keeps code compatible across all versions is that simple, just go with option 4. We document that in What's New and be done with it. :) -gps
On Jan 30, 2019, at 14:32, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
Now that regular dicts are ordered and compact, it makes more sense for the _asdict() method to create a regular dict (as it did in its early days) rather than an OrderedDict. The regular dict is much smaller, much faster, and has a much cleaner looking repr. It would also help namedtuple() stay in sync with dataclasses which already take advantage of the ordering feature of regular dicts.
Thanks for the excellent write up, Raymond.
Option 1) Add a deprecation notice to 3.8 […]
In general, I don’t favor deprecation notices for things that are not actionable by the end user or the consumer of the API in question. It ends up being just noise, and that’s one of the big reasons why they are silenced by default (i.e. the end user of a Python application can’t do anything about the deprecation, so why scare them?). Maybe we need something like a MinorBehavioralChangeWarning which would be surfaced similar to PendingDeprecationWarning, but which says “Hey, we changed this in the current version of Python. Be aware of that change, but you probably don’t need to do anything about it”. At least such warnings could be reviewed, documented, and audited. I suppose in this case, there is something the consumer of the API can do (as you point out, they can wrap their code), but in all likelihood there’s really nothing they *need* to do. I think it will be rare that a Python 3.8 user will get bitten by this. Therefore...
Option 4) Just make the change directly in 3.8,
JFDI! -Barry
On Wed, Jan 30, 2019 at 2:34 PM Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
Now that regular dicts are ordered and compact, it makes more sense for the _asdict() method to create a regular dict (as it did in its early days) rather than an OrderedDict. The regular dict is much smaller, much faster, and has a much cleaner looking repr.
How viable would it be to make OrderedDict smaller, faster, and give it a cleaner looking repr? -n -- Nathaniel J. Smith -- https://vorpus.org
On Jan 30, 2019, at 6:00 PM, David Mertz <mertz@gnosis.cx> wrote:
Ditto +1 option 4
On Wed, Jan 30, 2019, 5:56 PM Paul Moore <p.f.moore@gmail.com wrote: On Wed, 30 Jan 2019 at 22:35, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
My recommendation is Option 4 as being less disruptive and more beneficial than the other options. In the unlikely event that anyone is currently depending on the reordering methods for the output of _asdict(), the remediation is trivially simple: nt._asdict() -> OrderedDict(nt.as_dict()).
What do you all think?
+1 from me on option 4.
Paul
Thanks everyone. I'll move forward with option 4. In Barry's word, JFDI :-)
On Jan 30, 2019, at 6:10 PM, Nathaniel Smith <njs@pobox.com> wrote:
How viable would it be to make OrderedDict smaller, faster, and give it a cleaner looking repr?
Not so much. The implementations substantially different because they have different superpowers. A regular dict is really good at being a dict while retaining order but it isn't good at reordering operations such as popitem(False), popitem(True), move_to_end(), and whatnot. An OrderedDict is a heavier weight structure (a hash table augmented by a doubly-linked link) -- it is worse at being a dictionary but really good at intensive reordering operations typical in cache recency tracking and whatnot. Also, there are long-standing API differences including weak references, ability to assign attributes, an equality operation that requires exact order when compared to another ordered dict etc, as well as the reordering methods. If it was easy, clean, and desirable, it would have already been done :-) Overall, I think the OrderedDict is increasingly irrelevant except for use cases requiring cross-version compatibility and for cases that need heavy reordering. Accordingly, I mostly expect to leave it alone and fall into the not-much-used category like UserDict, UserList, and UserString.
On Jan 30, 2019, at 3:41 PM, Glenn Linderman <v+python@g.nevcal.com> wrote:
Would it be practical to add deprecated methods to regular dict for the OrderedDict reordering methods that raise with an error suggesting "To use this method, convert dict to OrderedDict." (or some better wording).
That's an interesting idea. Regular dicts aren't well suited to the reordering operations (like lists, repeated inserts at the front of the sequence wouldn't be performant relative to OrderedDict which uses double-linked lists internally). My instinct is to leave regular dicts alone so that they can focus on their primary task (being good a fast lookups). Raymond
On Thu, 31 Jan 2019 at 15:46, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
Would it be practical to add deprecated methods to regular dict for the OrderedDict reordering methods that raise with an error suggesting "To use this method, convert dict to OrderedDict." (or some better wording).
That's an interesting idea. Regular dicts aren't well suited to the reordering operations (like lists, repeated inserts at the front of the sequence wouldn't be performant relative to OrderedDict which uses double-linked lists internally). My instinct is to leave regular dicts alone so that they can focus on their primary task (being good a fast lookups).
Alternatively, would it be viable to make OrderedDict work in a way that so long as you don't use any reordering operations it's essentially just a very thin layer on top of a dict, but if you do use any reordering operations, it adds in the additional heavyweight structure required to support that? I'm pretty sure something similar has been considered before, but thought I should bring it up in the context of this discussion (if only to have to shot down). Tim Delaney
On Jan 30, 2019, at 9:11 PM, Tim Delaney <timothy.c.delaney@gmail.com> wrote:
Alternatively, would it be viable to make OrderedDict work in a way that so long as you don't use any reordering operations it's essentially just a very thin layer on top of a dict,
There's all kinds of tricks we could do but none of them are worth it. It took Eric Snow a long time to write the OrderedDict patch and it took years to get most of the bugs out of it. I would really hate to go through a redesign and eat up our time for something that probably won't be much used any more. I'm really just aiming for something as simple as s/OrderedDict/dict in namedtuple :-) Raymond
On 1/30/2019 8:45 PM, Raymond Hettinger wrote:
On Jan 30, 2019, at 3:41 PM, Glenn Linderman<v+python@g.nevcal.com> wrote: Would it be practical to add deprecated methods to regular dict for the OrderedDict reordering methods that raise with an error suggesting "To use this method, convert dict to OrderedDict." (or some better wording). That's an interesting idea. Regular dicts aren't well suited to the reordering operations (like lists, repeated inserts at the front of the sequence wouldn't be performant relative to OrderedDict which uses double-linked lists internally). My instinct is to leave regular dicts alone so that they can focus on their primary task (being good a fast lookups).
My goal was just to give a meaningful error message if someone misses the implications in What's New, and has code that actually does expect named_tuple.as_dict to have the ordering operations. The added, deprecated methods could be removed after a couple of versions.
On Thu, 31 Jan 2019 at 16:40, Glenn Linderman <v+python@g.nevcal.com> wrote:
On 1/30/2019 8:45 PM, Raymond Hettinger wrote:
On Jan 30, 2019, at 3:41 PM, Glenn Linderman <v+python@g.nevcal.com> wrote: Would it be practical to add deprecated methods to regular dict for the OrderedDict reordering methods that raise with an error suggesting "To use this method, convert dict to OrderedDict." (or some better wording). That's an interesting idea. Regular dicts aren't well suited to the reordering operations (like lists, repeated inserts at the front of the sequence wouldn't be performant relative to OrderedDict which uses double-linked lists internally). My instinct is to leave regular dicts alone so that they can focus on their primary task (being good a fast lookups). My goal was just to give a meaningful error message if someone misses the implications in What's New, and has code that actually does expect named_tuple.as_dict to have the ordering operations.
The downside of doing that is that automated code introspection tools don't know that the methods don't really exist, they just see the method names in the type dictionary and offer them up for code completion. So in this case, the extra runtime check isn't worth the cost of breaking static code analysis and other forms of introspection. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, Jan 31, 2019, 05:26 Nick Coghlan <ncoghlan@gmail.com wrote:
On 1/30/2019 8:45 PM, Raymond Hettinger wrote:
On Jan 30, 2019, at 3:41 PM, Glenn Linderman <v+python@g.nevcal.com> wrote: Would it be practical to add deprecated methods to regular dict for
That's an interesting idea. Regular dicts aren't well suited to the reordering operations (like lists, repeated inserts at the front of the sequence wouldn't be performant relative to OrderedDict which uses double-linked lists internally). My instinct is to leave regular dicts alone so that they can focus on their primary task (being good a fast lookups). My goal was just to give a meaningful error message if someone misses
On Thu, 31 Jan 2019 at 16:40, Glenn Linderman <v+python@g.nevcal.com> wrote: the OrderedDict reordering methods that raise with an error suggesting "To use this method, convert dict to OrderedDict." (or some better wording). the implications in What's New, and has code that actually does expect named_tuple.as_dict to have the ordering operations.
The downside of doing that is that automated code introspection tools don't know that the methods don't really exist, they just see the method names in the type dictionary and offer them up for code completion.
So in this case, the extra runtime check isn't worth the cost of breaking static code analysis and other forms of introspection.
It's technically possible for attributes to do something custom when accessed, without appearing in __dir__. I don't know if that's a useful technique in this case, but sometimes it is. -n
participants (11)
-
Antoine Pitrou -
Barry Warsaw -
David Mertz -
Ethan Furman -
Glenn Linderman -
Gregory P. Smith -
Nathaniel Smith -
Nick Coghlan -
Paul Moore -
Raymond Hettinger -
Tim Delaney