Add a dict with the attribute access capability
In 3.7 I have removed an old-deprecated plistlib.Dict. [1] Actually it already was deprecated when the plistlib module was added to the regular stdlib in Python 2.6. This is a dict subclass which allows to access items as attributes. d = plistlib.Dict() d['a'] = 1 assert d.a == 1 d.b = 2 assert d['b'] == 2 Raymond noticed that that capability seemed nice to have. What do you think about reviving this type as general purpose type in collections or types? Perhaps it can be convenient for working with JSON, plists, configuration files, databases and in other cases that need a dict with string keys. If reintroduce it, there are open questions. 1. The name of the type. 2. The location of the type. collections or types? Or other variants? 3. How it will collaborate with OrderedDict, defaultdict, etc? 4. Should it be a dict subclass, or a mixin, or a proxy? Or add several types? [1] https://bugs.python.org/issue29196
As reference of prior art, there is https://pypi.python.org/pypi/munch in PyPI On 29 November 2017 at 05:52, Serhiy Storchaka <storchaka@gmail.com> wrote:
-- Daniel F. Moisset - UK Country Manager - Machinalis Limited www.machinalis.co.uk <http://www.machinalis.com> Skype: @dmoisset T: + 44 7398 827139 1 Fore St, London, EC2Y 9DT Machinalis Limited is a company registered in England and Wales. Registered number: 10574987.
On Wed, Nov 29, 2017 at 7:21 AM, Daniel Moisset <dmoisset@machinalis.com> wrote:
As reference of prior art, there is https://pypi.python.org/pypi/munch in PyPI
Box is also popular as it permits deeper nesting: https://pypi.python.org/pypi/python-box/ https://github.com/cdgriffith/Box Addict is similar: https://pypi.python.org/pypi/addict/2.1.1 https://github.com/mewwts/addict and I've seen AttrDict a few times in code I maintain: https://pypi.python.org/pypi/attrdict https://github.com/bcj/AttrDict With a cursory search, also found: * DotMap: https://pypi.python.org/pypi/dotmap https://github.com/drgrib/dotmap * EasyDict https://pypi.python.org/pypi/easydict https://github.com/makinacorpus/easydict * Treedict (older) https://pypi.python.org/pypi/treedict * Bunch (older) https://pypi.python.org/pypi/bunch ...I spy a theme :P Haven't dug into them that much, but I'd try to answer most questions with how they do it. I'm not sure if the recursive 'items as attributes within items as attributes...' feature is a can of worms or if it's something like defaultdict(dict), but with more levels. Getting deeply nested items with dots seems to be the usual use (i.e. complicated JSONs), and I think the degree of laziness in creating/setting nested items is variable. Nick
On 29 November 2017 at 20:11, Barry Warsaw <barry@python.org> wrote:
This was discussed in https://github.com/ericvsmith/dataclasses/issues/21 and it was decided to postpone this. -- Ivan
On 30 November 2017 at 05:11, Barry Warsaw <barry@python.org> wrote:
Note that we do offer a simple namespace type: >>> from types import SimpleNamespace as ns >>> data = ns(a=1, b=2, c=3) >>> data.a 1 >>> vars(data)["a"] 1 >>> vars(data)["a"] = 3 >>> data.a 3 It was added as part of adding sys.implementation, since it's the storage type used for that: >>> import sys >>> type(sys.implementation) <class 'types.SimpleNamespace'> So the only thing we don't currently offer is a type that provides both attribute access and mapping access on the *same* object - for SimpleNamespace we require that you request the mapping API with "vars(ns)". Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Nov 29, 2017 at 12:52 AM, Serhiy Storchaka <storchaka@gmail.com> wrote:
What do you think about reviving this type as general purpose type in collections or types
Am I I the only one that thinks this is a "Bad Idea"? For me, it simply confuses even more the question of "is this code or is this data?" -- which is a difficult enough design question in a dynamic language. And the couple of libraries I"ve worked with that do this I liked at first, but grew to find problematic. The key problem is that keys in dicts can be anything immutable, while names have a lot more restrictions on them. And in a case like this, you can't use a key that happens to be the same as an existing dict attribute -- at least not in the same way. In particular, I've struggled with the netCDF4 lib and Colander. I've used netCDF4 more, so I'll talk about that: netCDF4 is a Python wrapper around the netcdf scientific data file format. netcdf has "variables" whiuch hold arrays of data, and attributes (for meta data). The netCDF4 python lib wraps a native netcdf4 variable with a python object that python attributes (names) for each of the netcdf variable attributes. Pretty nifty, really, you can do things like: vel = dataset.variables['velocity'] print("velocity is in units of:", vel.units) kinda cool, but in fact it gets harder to deal with in "real code" as opposed to quicky scripts than if variables has a dict-like attribute: print("velocity is in units of:", vel.atts['units']) Why? because netcdf variables can have arbitrary attributes, so your code tends to be hard-coded for specific data sets, and breaks fast if the data files are not compliant. That's what I mean by the difference between code and data -- the attributes of a netcdf variable is data -- not code. Granted, with Python, you can use try: except AttributeError, and setattr, and all that to work with these attributes as though they are data, but then you lose all the advantages of having them as attributes anyway. setattr and the like shulud be used for when metaprogramign is called for - not for everyday data manipulation. Oh, and it's also uglier to get/set values in bulk (though the proposed dict_with_attributes would present both APIs, so not as restrictive.) You also have problems with keys that mirror actual attributes. So you use the dict interface to deal with those -- but if you have to choose your interface according to what the values of the data are -- you've got the whole, it's not code, it's data! problem again. Anyway -- those are my $0.02 -- from experience working with such things in the real world, with real code, and real data.... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 11/29/2017 03:34 PM, Chris Barker wrote:
Am I I the only one that thinks this is a "Bad Idea"?
No, you're not.
For me, it simply confuses even more the question of "is this code or is this data?" -- which is a difficult enough design question in a dynamic language.
As one of those who have created their own version of AttrDict, I can say it's both neat and tricky to use. I'm happy to have it* as a third-party package, but I don't think it belongs in the stdlib. -- ~Ethan~ * attribute-access dicts in general, not mine in particular
On Wed, Nov 29, 2017 at 10:52:51AM +0200, Serhiy Storchaka wrote:
Isn't that a trivial subclass? class AttrDict(dict): def __init__(self, *args, **kw): super().__init__(*args, **kw) self.__dict__ = self Possibly apart from a nicer repr, have I missed anything?
I doubt that it is as convenient as you think. Most of the time I work with external data, I don't know the keys in advance, so I couldn't use obj.attr syntax. Configuration files (and equivalent) are possibly an exception. But in general I'm suspicious of using attribute access for key lookups. I cannot help feeling that, while laziness and hubris are virtues in programmers, this is *too* lazy. This conflates two distinct concepts, attributes and key/value pairs. Attributes are conceptually part of the object, as in dog.tail or car.engine, or data about the object, like page.top_margin or engine.capacity. Key/value data is not. Another way to look at this would be to say that deleting attributes is rare and unusual, while deleting keys is generally frequent and commonplace. Its not a completely hard distinction, I acknowledge that its already a bit fuzzy, but I'm not sure it is a good idea to make it even fuzzier. Consequently I'm strongly opposed to making this standard behaviour for dict and other mappings, but I'm neutral about making it opt-in for a distinct mapping type.
If reintroduce it, there are open questions.
1. The name of the type.
AttrDict seems like the obvious name.
2. The location of the type. collections or types? Or other variants?
It is a mapping, which is a collection, so collections.
3. How it will collaborate with OrderedDict, defaultdict, etc?
Does it need to?
4. Should it be a dict subclass, or a mixin, or a proxy? Or add several types?
Aren't these implementation details? -- Steve
I use argparse.Namespace whenever I need this. In reply to Chris Barker's concern of "is this code or is this data", the last time I used it is when I needed to hold an external function reference in an object instance (if I assigned it to an attribute, it was converted into an instance method). It just didn't feel right to invoke it via a container lookup. A telling name for the object also clears doubts about what it's supposed to hold. On 29.11.2017 11:52, Serhiy Storchaka wrote:
-- Regards, Ivan
On 01.12.2017 1:19, Greg Ewing wrote:
Well, yes, that was a singleton class, so I kept data in the class object. Now I can simplify the code by only keeping the instance reference in the class, thank you. (Without knowing this, that bore no visible benefits.) -- Regards, Ivan
As reference of prior art, there is https://pypi.python.org/pypi/munch in PyPI On 29 November 2017 at 05:52, Serhiy Storchaka <storchaka@gmail.com> wrote:
-- Daniel F. Moisset - UK Country Manager - Machinalis Limited www.machinalis.co.uk <http://www.machinalis.com> Skype: @dmoisset T: + 44 7398 827139 1 Fore St, London, EC2Y 9DT Machinalis Limited is a company registered in England and Wales. Registered number: 10574987.
On Wed, Nov 29, 2017 at 7:21 AM, Daniel Moisset <dmoisset@machinalis.com> wrote:
As reference of prior art, there is https://pypi.python.org/pypi/munch in PyPI
Box is also popular as it permits deeper nesting: https://pypi.python.org/pypi/python-box/ https://github.com/cdgriffith/Box Addict is similar: https://pypi.python.org/pypi/addict/2.1.1 https://github.com/mewwts/addict and I've seen AttrDict a few times in code I maintain: https://pypi.python.org/pypi/attrdict https://github.com/bcj/AttrDict With a cursory search, also found: * DotMap: https://pypi.python.org/pypi/dotmap https://github.com/drgrib/dotmap * EasyDict https://pypi.python.org/pypi/easydict https://github.com/makinacorpus/easydict * Treedict (older) https://pypi.python.org/pypi/treedict * Bunch (older) https://pypi.python.org/pypi/bunch ...I spy a theme :P Haven't dug into them that much, but I'd try to answer most questions with how they do it. I'm not sure if the recursive 'items as attributes within items as attributes...' feature is a can of worms or if it's something like defaultdict(dict), but with more levels. Getting deeply nested items with dots seems to be the usual use (i.e. complicated JSONs), and I think the degree of laziness in creating/setting nested items is variable. Nick
On 29 November 2017 at 20:11, Barry Warsaw <barry@python.org> wrote:
This was discussed in https://github.com/ericvsmith/dataclasses/issues/21 and it was decided to postpone this. -- Ivan
On 30 November 2017 at 05:11, Barry Warsaw <barry@python.org> wrote:
Note that we do offer a simple namespace type: >>> from types import SimpleNamespace as ns >>> data = ns(a=1, b=2, c=3) >>> data.a 1 >>> vars(data)["a"] 1 >>> vars(data)["a"] = 3 >>> data.a 3 It was added as part of adding sys.implementation, since it's the storage type used for that: >>> import sys >>> type(sys.implementation) <class 'types.SimpleNamespace'> So the only thing we don't currently offer is a type that provides both attribute access and mapping access on the *same* object - for SimpleNamespace we require that you request the mapping API with "vars(ns)". Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Nov 29, 2017 at 12:52 AM, Serhiy Storchaka <storchaka@gmail.com> wrote:
What do you think about reviving this type as general purpose type in collections or types
Am I I the only one that thinks this is a "Bad Idea"? For me, it simply confuses even more the question of "is this code or is this data?" -- which is a difficult enough design question in a dynamic language. And the couple of libraries I"ve worked with that do this I liked at first, but grew to find problematic. The key problem is that keys in dicts can be anything immutable, while names have a lot more restrictions on them. And in a case like this, you can't use a key that happens to be the same as an existing dict attribute -- at least not in the same way. In particular, I've struggled with the netCDF4 lib and Colander. I've used netCDF4 more, so I'll talk about that: netCDF4 is a Python wrapper around the netcdf scientific data file format. netcdf has "variables" whiuch hold arrays of data, and attributes (for meta data). The netCDF4 python lib wraps a native netcdf4 variable with a python object that python attributes (names) for each of the netcdf variable attributes. Pretty nifty, really, you can do things like: vel = dataset.variables['velocity'] print("velocity is in units of:", vel.units) kinda cool, but in fact it gets harder to deal with in "real code" as opposed to quicky scripts than if variables has a dict-like attribute: print("velocity is in units of:", vel.atts['units']) Why? because netcdf variables can have arbitrary attributes, so your code tends to be hard-coded for specific data sets, and breaks fast if the data files are not compliant. That's what I mean by the difference between code and data -- the attributes of a netcdf variable is data -- not code. Granted, with Python, you can use try: except AttributeError, and setattr, and all that to work with these attributes as though they are data, but then you lose all the advantages of having them as attributes anyway. setattr and the like shulud be used for when metaprogramign is called for - not for everyday data manipulation. Oh, and it's also uglier to get/set values in bulk (though the proposed dict_with_attributes would present both APIs, so not as restrictive.) You also have problems with keys that mirror actual attributes. So you use the dict interface to deal with those -- but if you have to choose your interface according to what the values of the data are -- you've got the whole, it's not code, it's data! problem again. Anyway -- those are my $0.02 -- from experience working with such things in the real world, with real code, and real data.... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 11/29/2017 03:34 PM, Chris Barker wrote:
Am I I the only one that thinks this is a "Bad Idea"?
No, you're not.
For me, it simply confuses even more the question of "is this code or is this data?" -- which is a difficult enough design question in a dynamic language.
As one of those who have created their own version of AttrDict, I can say it's both neat and tricky to use. I'm happy to have it* as a third-party package, but I don't think it belongs in the stdlib. -- ~Ethan~ * attribute-access dicts in general, not mine in particular
On Wed, Nov 29, 2017 at 10:52:51AM +0200, Serhiy Storchaka wrote:
Isn't that a trivial subclass? class AttrDict(dict): def __init__(self, *args, **kw): super().__init__(*args, **kw) self.__dict__ = self Possibly apart from a nicer repr, have I missed anything?
I doubt that it is as convenient as you think. Most of the time I work with external data, I don't know the keys in advance, so I couldn't use obj.attr syntax. Configuration files (and equivalent) are possibly an exception. But in general I'm suspicious of using attribute access for key lookups. I cannot help feeling that, while laziness and hubris are virtues in programmers, this is *too* lazy. This conflates two distinct concepts, attributes and key/value pairs. Attributes are conceptually part of the object, as in dog.tail or car.engine, or data about the object, like page.top_margin or engine.capacity. Key/value data is not. Another way to look at this would be to say that deleting attributes is rare and unusual, while deleting keys is generally frequent and commonplace. Its not a completely hard distinction, I acknowledge that its already a bit fuzzy, but I'm not sure it is a good idea to make it even fuzzier. Consequently I'm strongly opposed to making this standard behaviour for dict and other mappings, but I'm neutral about making it opt-in for a distinct mapping type.
If reintroduce it, there are open questions.
1. The name of the type.
AttrDict seems like the obvious name.
2. The location of the type. collections or types? Or other variants?
It is a mapping, which is a collection, so collections.
3. How it will collaborate with OrderedDict, defaultdict, etc?
Does it need to?
4. Should it be a dict subclass, or a mixin, or a proxy? Or add several types?
Aren't these implementation details? -- Steve
I use argparse.Namespace whenever I need this. In reply to Chris Barker's concern of "is this code or is this data", the last time I used it is when I needed to hold an external function reference in an object instance (if I assigned it to an attribute, it was converted into an instance method). It just didn't feel right to invoke it via a container lookup. A telling name for the object also clears doubts about what it's supposed to hold. On 29.11.2017 11:52, Serhiy Storchaka wrote:
-- Regards, Ivan
On 01.12.2017 1:19, Greg Ewing wrote:
Well, yes, that was a singleton class, so I kept data in the class object. Now I can simplify the code by only keeping the instance reference in the class, thank you. (Without knowing this, that bore no visible benefits.) -- Regards, Ivan
participants (12)
-
Barry Warsaw -
Chris Barker -
Daniel Moisset -
Ethan Furman -
Greg Ewing -
Ivan Levkivskyi -
Ivan Pozdeev -
Nick Coghlan -
Nick Timkovich -
Rob Cliffe -
Serhiy Storchaka -
Steven D'Aprano